[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781859658134510-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 09:00:58 dist-test-slave-mtdr krb5kdc[399](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 09:00:58 dist-test-slave-mtdr krb5kdc[399](info): set up 2 sockets
Jun 19 09:00:58 dist-test-slave-mtdr krb5kdc[399](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260619 09:00:59.094985 392 mini_kdc.cc:121] Time spent starting KDC: real 0.942s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781859658134510-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781859658134510-392-0/krb5kdc/test-user.keytab.
Jun 19 09:00:59 dist-test-slave-mtdr krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859659, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T09:00:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T09:00:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781859658134510-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781859658134510-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 09:00:59.205628 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ez5NB/build/release/bin/kudu
/tmp/dist-test-task4ez5NB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781859658134510-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781859658134510-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781859658134510-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781859658134510-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37585
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781859658134510-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41467
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37585
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781859658134510-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781859658134510-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781859658134510-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 09:00:59.282998 412 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 09:00:59.283202 412 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 09:00:59.283231 412 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 09:00:59.284358 412 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 09:00:59.284400 412 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 09:00:59.284426 412 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 09:00:59.284442 412 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 09:00:59.284458 412 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 09:00:59.285900 412 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41467
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781859658134510-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781859658134510-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37585
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781859658134510-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37585
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781859658134510-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.412
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781859658134510-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 09:00:59.286162 412 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 09:00:59.286379 412 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 09:00:59.291152 420 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:00:59.291129 418 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:00:59.291131 417 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:00:59.291254 412 server_base.cc:1034] running on GCE node
I20260619 09:00:59.291594 412 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 09:00:59.291842 412 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 09:00:59.293004 412 hybrid_clock.cc:648] HybridClock initialized: now 1781859659292978 us; error 44 us; skew 500 ppm
Jun 19 09:00:59 dist-test-slave-mtdr krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859659, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 09:00:59.295423 412 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 09:00:59.295885 412 webserver.cc:460] Webserver started at http://127.0.98.62:46381/ using document root <none> and password file <none>
I20260619 09:00:59.296096 412 fs_manager.cc:359] Metadata directory not provided
I20260619 09:00:59.296144 412 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 09:00:59.296267 412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 09:00:59.297288 412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781859658134510-392-0/minicluster-data/master-0/data/instance:
uuid: "280b460f3edc46f393732fddca88c74d"
format_stamp: "Formatted at 2026-06-19 09:00:59 on dist-test-slave-mtdr"
server_key: "de0145aa54936a48feb24578555641b7"
server_key_iv: "b2772d7c29e644109a501806ce748aca"
server_key_version: "encryptionkey@0"
I20260619 09:00:59.297622 412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781859658134510-392-0/minicluster-data/master-0/wal/instance:
uuid: "280b460f3edc46f393732fddca88c74d"
format_stamp: "Formatted at 2026-06-19 09:00:59 on dist-test-slave-mtdr"
server_key: "de0145aa54936a48feb24578555641b7"
server_key_iv: "b2772d7c29e644109a501806ce748aca"
server_key_version: "encryptionkey@0"
I20260619 09:00:59.298897 412 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260619 09:00:59.299623 427 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 09:00:59.299787 412 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 09:00:59.299866 412 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781859658134510-392-0/minicluster-data/master-0/data,/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781859658134510-392-0/minicluster-data/master-0/wal
uuid: "280b460f3edc46f393732fddca88c74d"
format_stamp: "Formatted at 2026-06-19 09:00:59 on dist-test-slave-mtdr"
server_key: "de0145aa54936a48feb24578555641b7"
server_key_iv: "b2772d7c29e644109a501806ce748aca"
server_key_version: "encryptionkey@0"
I20260619 09:00:59.299932 412 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781859658134510-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781859658134510-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781859658134510-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 09:00:59.317296 412 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 09:00:59.319092 412 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 09:00:59.319226 412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 09:00:59.322696 412 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37585
I20260619 09:00:59.322744 479 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37585 every 8 connection(s)
I20260619 09:00:59.323060 412 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781859658134510-392-0/minicluster-data/master-0/data/info.pb
I20260619 09:00:59.323889 480 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 09:00:59.326167 480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 280b460f3edc46f393732fddca88c74d: Bootstrap starting.
I20260619 09:00:59.326733 480 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 280b460f3edc46f393732fddca88c74d: Neither blocks nor log segments found. Creating new log.
I20260619 09:00:59.327026 480 log.cc:824] T 00000000000000000000000000000000 P 280b460f3edc46f393732fddca88c74d: Log is configured to *not* fsync() on all Append() calls
I20260619 09:00:59.327680 480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 280b460f3edc46f393732fddca88c74d: No bootstrap required, opened a new log
I20260619 09:00:59.329258 480 raft_consensus.cc:348] T 00000000000000000000000000000000 P 280b460f3edc46f393732fddca88c74d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "280b460f3edc46f393732fddca88c74d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37585 } }
I20260619 09:00:59.329406 480 raft_consensus.cc:374] T 00000000000000000000000000000000 P 280b460f3edc46f393732fddca88c74d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 09:00:59.329430 480 raft_consensus.cc:729] T 00000000000000000000000000000000 P 280b460f3edc46f393732fddca88c74d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 280b460f3edc46f393732fddca88c74d, State: Initialized, Role: FOLLOWER
I20260619 09:00:59.329574 480 consensus_queue.cc:260] T 00000000000000000000000000000000 P 280b460f3edc46f393732fddca88c74d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "280b460f3edc46f393732fddca88c74d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37585 } }
I20260619 09:00:59.329643 480 raft_consensus.cc:388] T 00000000000000000000000000000000 P 280b460f3edc46f393732fddca88c74d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 09:00:59.329676 480 raft_consensus.cc:482] T 00000000000000000000000000000000 P 280b460f3edc46f393732fddca88c74d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 09:00:59.329728 480 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 280b460f3edc46f393732fddca88c74d [term 0 FOLLOWER]: Advancing to term 1
I20260619 09:00:59.329866 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ez5NB/build/release/bin/kudu as pid 412
I20260619 09:00:59.329947 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781859658134510-392-0/minicluster-data/master-0/wal/instance
I20260619 09:00:59.330294 480 raft_consensus.cc:504] T 00000000000000000000000000000000 P 280b460f3edc46f393732fddca88c74d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "280b460f3edc46f393732fddca88c74d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37585 } }
I20260619 09:00:59.330394 480 leader_election.cc:302] T 00000000000000000000000000000000 P 280b460f3edc46f393732fddca88c74d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 280b460f3edc46f393732fddca88c74d; no voters:
I20260619 09:00:59.330554 480 leader_election.cc:288] T 00000000000000000000000000000000 P 280b460f3edc46f393732fddca88c74d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 09:00:59.330577 392 external_mini_cluster.cc:1428] Setting key f42b6f807eb94062d4986f527f7c6b9d
I20260619 09:00:59.330595 483 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 280b460f3edc46f393732fddca88c74d [term 1 FOLLOWER]: Leader election won for term 1
I20260619 09:00:59.330752 483 raft_consensus.cc:686] T 00000000000000000000000000000000 P 280b460f3edc46f393732fddca88c74d [term 1 LEADER]: Becoming Leader. State: Replica: 280b460f3edc46f393732fddca88c74d, State: Running, Role: LEADER
I20260619 09:00:59.330806 480 sys_catalog.cc:564] T 00000000000000000000000000000000 P 280b460f3edc46f393732fddca88c74d [sys.catalog]: configured and running, proceeding with master startup.
I20260619 09:00:59.330988 483 consensus_queue.cc:237] T 00000000000000000000000000000000 P 280b460f3edc46f393732fddca88c74d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "280b460f3edc46f393732fddca88c74d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37585 } }
I20260619 09:00:59.332038 485 sys_catalog.cc:455] T 00000000000000000000000000000000 P 280b460f3edc46f393732fddca88c74d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "280b460f3edc46f393732fddca88c74d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "280b460f3edc46f393732fddca88c74d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37585 } } }
I20260619 09:00:59.332093 484 sys_catalog.cc:455] T 00000000000000000000000000000000 P 280b460f3edc46f393732fddca88c74d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 280b460f3edc46f393732fddca88c74d. Latest consensus state: current_term: 1 leader_uuid: "280b460f3edc46f393732fddca88c74d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "280b460f3edc46f393732fddca88c74d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37585 } } }
I20260619 09:00:59.332208 484 sys_catalog.cc:458] T 00000000000000000000000000000000 P 280b460f3edc46f393732fddca88c74d [sys.catalog]: This master's current role is: LEADER
I20260619 09:00:59.332316 485 sys_catalog.cc:458] T 00000000000000000000000000000000 P 280b460f3edc46f393732fddca88c74d [sys.catalog]: This master's current role is: LEADER
W20260619 09:00:59.332894 500 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 280b460f3edc46f393732fddca88c74d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 09:00:59.332944 500 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 09:00:59.333094 501 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 09:00:59.333240 501 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 09:00:59.334653 501 catalog_manager.cc:1347] Generated new cluster ID: 41d88c367eb54538bd2d79763d575bc6
I20260619 09:00:59.334702 501 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 19 09:00:59 dist-test-slave-mtdr krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859659, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 09:00:59.344966 496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:00:59.331890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47924 (local address 127.0.98.62:37585)
0619 09:00:59.332226 (+ 336us) server_negotiation.cc:206] Beginning negotiation
0619 09:00:59.332235 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 09:00:59.332269 (+ 34us) server_negotiation.cc:407] Connection header received
0619 09:00:59.334043 (+ 1774us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:00:59.334050 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:00:59.334100 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:00:59.334232 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:00:59.335834 (+ 1602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:00:59.336763 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:00:59.337778 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:00:59.338197 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:00:59.338298 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:00:59.341526 (+ 3228us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:00:59.341531 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:00:59.341532 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:00:59.341543 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:00:59.343233 (+ 1690us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:00:59.343629 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:00:59.343632 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:00:59.343633 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:00:59.343722 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:00:59.343956 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:00:59.343956 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:00:59.343957 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:00:59.344180 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:00:59.344329 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0619 09:00:59.344588 (+ 259us) server_negotiation.cc:299] Negotiation successful
0619 09:00:59.344713 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":291,"thread_start_us":84,"threads_started":1}
I20260619 09:00:59.346045 501 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 09:00:59.346702 501 catalog_manager.cc:1504] Loading token signing keys...
I20260619 09:00:59.357784 501 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 280b460f3edc46f393732fddca88c74d: Generated new TSK 0
I20260619 09:00:59.357966 501 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781859658134510-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781859658134510-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 09:00:59.383131 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ez5NB/build/release/bin/kudu
/tmp/dist-test-task4ez5NB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781859658134510-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781859658134510-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781859658134510-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781859658134510-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37585
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781859658134510-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41467
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781859658134510-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781859658134510-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781859658134510-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 09:00:59.460389 506 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 09:00:59.460605 506 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 09:00:59.460637 506 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 09:00:59.461740 506 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 09:00:59.461776 506 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 09:00:59.461812 506 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 09:00:59.463222 506 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41467
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781859658134510-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781859658134510-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781859658134510-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781859658134510-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37585
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.506
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781859658134510-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 09:00:59.463447 506 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 09:00:59.463646 506 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 09:00:59.468116 511 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:00:59.468182 514 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:00:59.468191 512 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:00:59.468286 506 server_base.cc:1034] running on GCE node
I20260619 09:00:59.468523 506 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 09:00:59.468748 506 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 09:00:59.469887 506 hybrid_clock.cc:648] HybridClock initialized: now 1781859659469869 us; error 33 us; skew 500 ppm
Jun 19 09:00:59 dist-test-slave-mtdr krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859659, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 09:00:59.472167 506 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 09:00:59.472558 506 webserver.cc:460] Webserver started at http://127.0.98.1:33371/ using document root <none> and password file <none>
I20260619 09:00:59.472754 506 fs_manager.cc:359] Metadata directory not provided
I20260619 09:00:59.472796 506 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 09:00:59.472903 506 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 09:00:59.473878 506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781859658134510-392-0/minicluster-data/ts-0/data/instance:
uuid: "9e90f473abd942d7a7f332a756ccdda5"
format_stamp: "Formatted at 2026-06-19 09:00:59 on dist-test-slave-mtdr"
server_key: "c1d2551aebf966fb5a2692c58627741f"
server_key_iv: "ef40ef5e621bba0d4a1da63365a6127a"
server_key_version: "encryptionkey@0"
I20260619 09:00:59.474189 506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781859658134510-392-0/minicluster-data/ts-0/wal/instance:
uuid: "9e90f473abd942d7a7f332a756ccdda5"
format_stamp: "Formatted at 2026-06-19 09:00:59 on dist-test-slave-mtdr"
server_key: "c1d2551aebf966fb5a2692c58627741f"
server_key_iv: "ef40ef5e621bba0d4a1da63365a6127a"
server_key_version: "encryptionkey@0"
I20260619 09:00:59.475484 506 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 09:00:59.476184 521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 09:00:59.476351 506 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 09:00:59.476432 506 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781859658134510-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781859658134510-392-0/minicluster-data/ts-0/wal
uuid: "9e90f473abd942d7a7f332a756ccdda5"
format_stamp: "Formatted at 2026-06-19 09:00:59 on dist-test-slave-mtdr"
server_key: "c1d2551aebf966fb5a2692c58627741f"
server_key_iv: "ef40ef5e621bba0d4a1da63365a6127a"
server_key_version: "encryptionkey@0"
I20260619 09:00:59.476492 506 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781859658134510-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781859658134510-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781859658134510-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 09:00:59.486811 506 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 09:00:59.488806 506 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 09:00:59.488931 506 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 09:00:59.489122 506 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 09:00:59.489405 506 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 09:00:59.489444 506 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 09:00:59.489464 506 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 09:00:59.489488 506 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 09:00:59.494761 506 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:33097
I20260619 09:00:59.494841 634 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:33097 every 8 connection(s)
I20260619 09:00:59.495126 506 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781859658134510-392-0/minicluster-data/ts-0/data/info.pb
I20260619 09:00:59.497292 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ez5NB/build/release/bin/kudu as pid 506
I20260619 09:00:59.497365 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781859658134510-392-0/minicluster-data/ts-0/wal/instance
I20260619 09:00:59.497488 392 external_mini_cluster.cc:1428] Setting key ebf87f30c1d34cd1700cb8efac0d5e35
Jun 19 09:00:59 dist-test-slave-mtdr krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859659, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 09:00:59.504017 496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:00:59.495791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56415 (local address 127.0.98.62:37585)
0619 09:00:59.495848 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0619 09:00:59.495850 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:00:59.496051 (+ 201us) server_negotiation.cc:407] Connection header received
0619 09:00:59.496548 (+ 497us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:00:59.496549 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:00:59.496562 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:00:59.496658 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:00:59.497575 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:00:59.498252 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:00:59.499279 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:00:59.499485 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:00:59.499542 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:00:59.501459 (+ 1917us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:00:59.501460 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:00:59.501461 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:00:59.501467 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:00:59.502751 (+ 1284us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:00:59.503117 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:00:59.503118 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:00:59.503118 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:00:59.503175 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:00:59.503408 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:00:59.503408 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:00:59.503409 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:00:59.503527 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:00:59.503583 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0619 09:00:59.503845 (+ 262us) server_negotiation.cc:299] Negotiation successful
0619 09:00:59.503930 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260619 09:00:59.504199 637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:00:59.495796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37585 (local address 127.0.98.1:56415)
0619 09:00:59.495934 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0619 09:00:59.495968 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0619 09:00:59.496442 (+ 474us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:00:59.496726 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:00:59.496729 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:00:59.497263 (+ 534us) client_negotiation.cc:189] Negotiated authn=SASL
0619 09:00:59.497485 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:00:59.497490 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:00:59.498340 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:00:59.498341 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:00:59.499203 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:00:59.499206 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:00:59.499553 (+ 347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:00:59.499553 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:00:59.499672 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:00:59.500119 (+ 447us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 09:00:59.500127 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 09:00:59.501374 (+ 1247us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 09:00:59.502843 (+ 1469us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:00:59.502849 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:00:59.502852 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:00:59.503053 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:00:59.503234 (+ 181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:00:59.503235 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:00:59.503235 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:00:59.503354 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:00:59.503582 (+ 228us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 09:00:59.503589 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 09:00:59.503726 (+ 137us) client_negotiation.cc:769] Sending connection context
0619 09:00:59.503824 (+ 98us) client_negotiation.cc:240] Negotiation successful
0619 09:00:59.503971 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":72,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260619 09:00:59.504655 635 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37585
I20260619 09:00:59.504741 635 heartbeater.cc:464] Registering TS with master...
I20260619 09:00:59.504904 635 heartbeater.cc:511] Master 127.0.98.62:37585 requested a full tablet report, sending...
I20260619 09:00:59.505357 444 ts_manager.cc:194] Registered new tserver with Master: 9e90f473abd942d7a7f332a756ccdda5 (127.0.98.1:33097)
I20260619 09:00:59.505970 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:56415
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781859658134510-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781859658134510-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 09:00:59.521072 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ez5NB/build/release/bin/kudu
/tmp/dist-test-task4ez5NB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781859658134510-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781859658134510-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781859658134510-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781859658134510-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37585
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781859658134510-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41467
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781859658134510-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781859658134510-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781859658134510-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 09:00:59.598634 640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 09:00:59.598853 640 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 09:00:59.598877 640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 09:00:59.599957 640 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 09:00:59.599989 640 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 09:00:59.600013 640 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 09:00:59.601347 640 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41467
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781859658134510-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781859658134510-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781859658134510-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781859658134510-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37585
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.640
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781859658134510-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 09:00:59.601518 640 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 09:00:59.601692 640 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 09:00:59.606542 645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:00:59.606642 640 server_base.cc:1034] running on GCE node
W20260619 09:00:59.606658 648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:00:59.606762 646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:00:59.606993 640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 09:00:59.607227 640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 09:00:59.608376 640 hybrid_clock.cc:648] HybridClock initialized: now 1781859659608361 us; error 29 us; skew 500 ppm
Jun 19 09:00:59 dist-test-slave-mtdr krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859659, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 09:00:59.610807 640 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 09:00:59.611233 640 webserver.cc:460] Webserver started at http://127.0.98.2:33921/ using document root <none> and password file <none>
I20260619 09:00:59.611438 640 fs_manager.cc:359] Metadata directory not provided
I20260619 09:00:59.611483 640 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 09:00:59.611598 640 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 09:00:59.612609 640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781859658134510-392-0/minicluster-data/ts-1/data/instance:
uuid: "14677755cfef428d954ef792b44943d9"
format_stamp: "Formatted at 2026-06-19 09:00:59 on dist-test-slave-mtdr"
server_key: "f5c709c63dfaa3e9cd6fab13cb271bb4"
server_key_iv: "ec8c31a8ed349ac56e2430cc6abd579f"
server_key_version: "encryptionkey@0"
I20260619 09:00:59.612936 640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781859658134510-392-0/minicluster-data/ts-1/wal/instance:
uuid: "14677755cfef428d954ef792b44943d9"
format_stamp: "Formatted at 2026-06-19 09:00:59 on dist-test-slave-mtdr"
server_key: "f5c709c63dfaa3e9cd6fab13cb271bb4"
server_key_iv: "ec8c31a8ed349ac56e2430cc6abd579f"
server_key_version: "encryptionkey@0"
I20260619 09:00:59.614110 640 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 09:00:59.614701 655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 09:00:59.614881 640 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 09:00:59.614952 640 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781859658134510-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781859658134510-392-0/minicluster-data/ts-1/wal
uuid: "14677755cfef428d954ef792b44943d9"
format_stamp: "Formatted at 2026-06-19 09:00:59 on dist-test-slave-mtdr"
server_key: "f5c709c63dfaa3e9cd6fab13cb271bb4"
server_key_iv: "ec8c31a8ed349ac56e2430cc6abd579f"
server_key_version: "encryptionkey@0"
I20260619 09:00:59.615016 640 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781859658134510-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781859658134510-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781859658134510-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 09:00:59.630262 640 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 09:00:59.632099 640 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 09:00:59.632241 640 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 09:00:59.632437 640 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 09:00:59.632721 640 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 09:00:59.632762 640 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 09:00:59.632797 640 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 09:00:59.632820 640 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 09:00:59.638322 640 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33913
I20260619 09:00:59.638361 768 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33913 every 8 connection(s)
I20260619 09:00:59.638639 640 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781859658134510-392-0/minicluster-data/ts-1/data/info.pb
Jun 19 09:00:59 dist-test-slave-mtdr krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859659, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 09:00:59.645531 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ez5NB/build/release/bin/kudu as pid 640
I20260619 09:00:59.645646 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781859658134510-392-0/minicluster-data/ts-1/wal/instance
I20260619 09:00:59.645818 392 external_mini_cluster.cc:1428] Setting key dfed23ec17d089c3e7458139e10d319e
I20260619 09:00:59.646957 496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:00:59.639285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39641 (local address 127.0.98.62:37585)
0619 09:00:59.639335 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0619 09:00:59.639337 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:00:59.639529 (+ 192us) server_negotiation.cc:407] Connection header received
0619 09:00:59.639981 (+ 452us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:00:59.639983 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:00:59.639992 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:00:59.640044 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:00:59.640909 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:00:59.641391 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:00:59.642352 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:00:59.642539 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:00:59.642595 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:00:59.644455 (+ 1860us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:00:59.644456 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:00:59.644456 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:00:59.644462 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:00:59.645495 (+ 1033us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:00:59.645932 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:00:59.645934 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:00:59.645934 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:00:59.646003 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:00:59.646267 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:00:59.646268 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:00:59.646268 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:00:59.646398 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:00:59.646464 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0619 09:00:59.646796 (+ 332us) server_negotiation.cc:299] Negotiation successful
0619 09:00:59.646897 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260619 09:00:59.647177 771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:00:59.639269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37585 (local address 127.0.98.2:39641)
0619 09:00:59.639418 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0619 09:00:59.639459 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 09:00:59.639899 (+ 440us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:00:59.640112 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:00:59.640116 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:00:59.640646 (+ 530us) client_negotiation.cc:189] Negotiated authn=SASL
0619 09:00:59.640829 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:00:59.640833 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:00:59.641468 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:00:59.641468 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:00:59.642259 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:00:59.642261 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:00:59.642603 (+ 342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:00:59.642603 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:00:59.642694 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:00:59.643145 (+ 451us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 09:00:59.643153 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 09:00:59.644381 (+ 1228us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 09:00:59.645562 (+ 1181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:00:59.645568 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:00:59.645570 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:00:59.645831 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:00:59.646081 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:00:59.646082 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:00:59.646083 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:00:59.646202 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:00:59.646490 (+ 288us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 09:00:59.646497 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 09:00:59.646656 (+ 159us) client_negotiation.cc:769] Sending connection context
0619 09:00:59.646781 (+ 125us) client_negotiation.cc:240] Negotiation successful
0619 09:00:59.646952 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":68,"threads_started":1}
I20260619 09:00:59.648140 769 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37585
I20260619 09:00:59.648221 769 heartbeater.cc:464] Registering TS with master...
I20260619 09:00:59.648404 769 heartbeater.cc:511] Master 127.0.98.62:37585 requested a full tablet report, sending...
I20260619 09:00:59.648800 444 ts_manager.cc:194] Registered new tserver with Master: 14677755cfef428d954ef792b44943d9 (127.0.98.2:33913)
I20260619 09:00:59.649155 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:39641
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781859658134510-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781859658134510-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 09:00:59.667775 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ez5NB/build/release/bin/kudu
/tmp/dist-test-task4ez5NB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781859658134510-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781859658134510-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781859658134510-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781859658134510-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37585
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781859658134510-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41467
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781859658134510-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781859658134510-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781859658134510-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 09:00:59.746177 774 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 09:00:59.746380 774 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 09:00:59.746410 774 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 09:00:59.747560 774 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 09:00:59.747598 774 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 09:00:59.747632 774 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 09:00:59.749001 774 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41467
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781859658134510-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781859658134510-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781859658134510-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781859658134510-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37585
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.774
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781859658134510-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 09:00:59.749224 774 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 09:00:59.749428 774 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 09:00:59.754047 782 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:00:59.754060 779 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:00:59.754081 780 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:00:59.754371 774 server_base.cc:1034] running on GCE node
I20260619 09:00:59.754570 774 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 09:00:59.754789 774 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 09:00:59.755966 774 hybrid_clock.cc:648] HybridClock initialized: now 1781859659755950 us; error 28 us; skew 500 ppm
Jun 19 09:00:59 dist-test-slave-mtdr krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859659, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 09:00:59.758347 774 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 09:00:59.758729 774 webserver.cc:460] Webserver started at http://127.0.98.3:38919/ using document root <none> and password file <none>
I20260619 09:00:59.758962 774 fs_manager.cc:359] Metadata directory not provided
I20260619 09:00:59.759009 774 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 09:00:59.759111 774 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 09:00:59.760074 774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781859658134510-392-0/minicluster-data/ts-2/data/instance:
uuid: "a54b9af0b3f1412386455e7debf44512"
format_stamp: "Formatted at 2026-06-19 09:00:59 on dist-test-slave-mtdr"
server_key: "46a8607d148dac4a41e38e542ee9a76d"
server_key_iv: "5ffcd2d5ddaec00e188cda22e4603f4a"
server_key_version: "encryptionkey@0"
I20260619 09:00:59.760373 774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781859658134510-392-0/minicluster-data/ts-2/wal/instance:
uuid: "a54b9af0b3f1412386455e7debf44512"
format_stamp: "Formatted at 2026-06-19 09:00:59 on dist-test-slave-mtdr"
server_key: "46a8607d148dac4a41e38e542ee9a76d"
server_key_iv: "5ffcd2d5ddaec00e188cda22e4603f4a"
server_key_version: "encryptionkey@0"
I20260619 09:00:59.761577 774 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 09:00:59.762207 789 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 09:00:59.762353 774 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 09:00:59.762425 774 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781859658134510-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781859658134510-392-0/minicluster-data/ts-2/wal
uuid: "a54b9af0b3f1412386455e7debf44512"
format_stamp: "Formatted at 2026-06-19 09:00:59 on dist-test-slave-mtdr"
server_key: "46a8607d148dac4a41e38e542ee9a76d"
server_key_iv: "5ffcd2d5ddaec00e188cda22e4603f4a"
server_key_version: "encryptionkey@0"
I20260619 09:00:59.762470 774 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781859658134510-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781859658134510-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781859658134510-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 09:00:59.785756 774 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 09:00:59.787694 774 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 09:00:59.787801 774 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 09:00:59.787982 774 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 09:00:59.788226 774 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 09:00:59.788254 774 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 09:00:59.788275 774 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 09:00:59.788290 774 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 09:00:59.794394 774 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44689
I20260619 09:00:59.794469 902 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44689 every 8 connection(s)
I20260619 09:00:59.794747 774 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781859658134510-392-0/minicluster-data/ts-2/data/info.pb
Jun 19 09:00:59 dist-test-slave-mtdr krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859659, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 09:00:59.802002 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ez5NB/build/release/bin/kudu as pid 774
I20260619 09:00:59.802083 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781859658134510-392-0/minicluster-data/ts-2/wal/instance
I20260619 09:00:59.802235 392 external_mini_cluster.cc:1428] Setting key 6c824a573ea786606bc9a47e04c38d47
I20260619 09:00:59.803365 496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:00:59.795431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44469 (local address 127.0.98.62:37585)
0619 09:00:59.795504 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0619 09:00:59.795506 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:00:59.795664 (+ 158us) server_negotiation.cc:407] Connection header received
0619 09:00:59.796119 (+ 455us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:00:59.796120 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:00:59.796130 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:00:59.796174 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:00:59.797053 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:00:59.797530 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:00:59.798451 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:00:59.798620 (+ 169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:00:59.798661 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:00:59.800613 (+ 1952us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:00:59.800614 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:00:59.800615 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:00:59.800620 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:00:59.802036 (+ 1416us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:00:59.802456 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:00:59.802458 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:00:59.802458 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:00:59.802512 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:00:59.802729 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:00:59.802730 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:00:59.802730 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:00:59.802880 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:00:59.802956 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0619 09:00:59.803208 (+ 252us) server_negotiation.cc:299] Negotiation successful
0619 09:00:59.803281 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260619 09:00:59.803478 905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:00:59.795415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37585 (local address 127.0.98.3:44469)
0619 09:00:59.795557 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0619 09:00:59.795591 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0619 09:00:59.796044 (+ 453us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:00:59.796236 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:00:59.796239 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:00:59.796799 (+ 560us) client_negotiation.cc:189] Negotiated authn=SASL
0619 09:00:59.796988 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:00:59.796992 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:00:59.797595 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:00:59.797596 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:00:59.798393 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:00:59.798395 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:00:59.798670 (+ 275us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:00:59.798671 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:00:59.798764 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:00:59.799228 (+ 464us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 09:00:59.799236 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 09:00:59.800543 (+ 1307us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 09:00:59.802129 (+ 1586us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:00:59.802136 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:00:59.802138 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:00:59.802384 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:00:59.802563 (+ 179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:00:59.802564 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:00:59.802564 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:00:59.802672 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:00:59.802953 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 09:00:59.802959 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 09:00:59.803088 (+ 129us) client_negotiation.cc:769] Sending connection context
0619 09:00:59.803180 (+ 92us) client_negotiation.cc:240] Negotiation successful
0619 09:00:59.803308 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":73,"threads_started":1}
I20260619 09:00:59.803853 903 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37585
I20260619 09:00:59.803934 903 heartbeater.cc:464] Registering TS with master...
I20260619 09:00:59.804090 903 heartbeater.cc:511] Master 127.0.98.62:37585 requested a full tablet report, sending...
I20260619 09:00:59.804399 444 ts_manager.cc:194] Registered new tserver with Master: a54b9af0b3f1412386455e7debf44512 (127.0.98.3:44689)
I20260619 09:00:59.804734 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:44469
I20260619 09:00:59.814116 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 19 09:00:59 dist-test-slave-mtdr krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859659, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 09:00:59 dist-test-slave-mtdr krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859659, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260619 09:00:59.835966 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:00:59.828353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47914 (local address 127.0.98.1:33097)
0619 09:00:59.828508 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0619 09:00:59.828513 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 09:00:59.828523 (+ 10us) server_negotiation.cc:407] Connection header received
0619 09:00:59.828546 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:00:59.828548 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:00:59.828562 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:00:59.828624 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:00:59.829095 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:00:59.829795 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:00:59.830530 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:00:59.830939 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:00:59.831008 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:00:59.833248 (+ 2240us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:00:59.833256 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:00:59.833256 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:00:59.833264 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:00:59.834789 (+ 1525us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:00:59.835179 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:00:59.835183 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:00:59.835183 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:00:59.835243 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:00:59.835425 (+ 182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:00:59.835426 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:00:59.835426 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:00:59.835560 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:00:59.835706 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0619 09:00:59.835778 (+ 72us) server_negotiation.cc:299] Negotiation successful
0619 09:00:59.835853 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":87,"threads_started":1}
W20260619 09:00:59.836728 549 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47914
Jun 19 09:00:59 dist-test-slave-mtdr krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859659, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 09:00:59 dist-test-slave-mtdr krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859659, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260619 09:00:59.856387 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:00:59.850136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47928 (local address 127.0.98.1:33097)
0619 09:00:59.850186 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0619 09:00:59.850188 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:00:59.850256 (+ 68us) server_negotiation.cc:407] Connection header received
0619 09:00:59.850318 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:00:59.850319 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:00:59.850329 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:00:59.850374 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:00:59.850884 (+ 510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:00:59.851362 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:00:59.852046 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:00:59.852265 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:00:59.852319 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:00:59.854267 (+ 1948us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:00:59.854268 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:00:59.854269 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:00:59.854275 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:00:59.855471 (+ 1196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:00:59.855812 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:00:59.855812 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:00:59.855813 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:00:59.855853 (+ 40us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:00:59.855998 (+ 145us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:00:59.855998 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:00:59.855998 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:00:59.856073 (+ 75us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:00:59.856150 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0619 09:00:59.856270 (+ 120us) server_negotiation.cc:299] Negotiation successful
0619 09:00:59.856325 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260619 09:00:59.856818 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ez5NB/build/release/bin/kudu with pid 506
I20260619 09:00:59.860893 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ez5NB/build/release/bin/kudu with pid 640
I20260619 09:00:59.865509 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ez5NB/build/release/bin/kudu with pid 774
I20260619 09:00:59.868863 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ez5NB/build/release/bin/kudu with pid 412
2026-06-19T09:00:59Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (1739 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781859658134510-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 09:00:59 dist-test-slave-mtdr krb5kdc[917](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 09:00:59 dist-test-slave-mtdr krb5kdc[917](info): set up 2 sockets
Jun 19 09:00:59 dist-test-slave-mtdr krb5kdc[917](info): commencing operation
krb5kdc: starting...
W20260619 09:01:00.818168 392 mini_kdc.cc:121] Time spent starting KDC: real 0.931s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781859658134510-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781859658134510-392-0/krb5kdc/test-user.keytab.
Jun 19 09:01:00 dist-test-slave-mtdr krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859660, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T09:01:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T09:01:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781859658134510-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781859658134510-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 09:01:00.927528 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ez5NB/build/release/bin/kudu
/tmp/dist-test-task4ez5NB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781859658134510-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781859658134510-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781859658134510-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781859658134510-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37287
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781859658134510-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37075
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37287
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781859658134510-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781859658134510-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781859658134510-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 09:01:01.005374 930 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 09:01:01.005576 930 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 09:01:01.005597 930 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 09:01:01.006736 930 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 09:01:01.006779 930 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 09:01:01.006793 930 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 09:01:01.006803 930 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 09:01:01.006843 930 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 09:01:01.008302 930 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37075
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781859658134510-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781859658134510-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37287
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781859658134510-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37287
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781859658134510-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.930
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781859658134510-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 09:01:01.008522 930 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 09:01:01.008738 930 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 09:01:01.013285 935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:01:01.013298 936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:01:01.013348 938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:01:01.013432 930 server_base.cc:1034] running on GCE node
I20260619 09:01:01.013733 930 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 09:01:01.013988 930 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 09:01:01.015142 930 hybrid_clock.cc:648] HybridClock initialized: now 1781859661015129 us; error 37 us; skew 500 ppm
Jun 19 09:01:01 dist-test-slave-mtdr krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859661, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 09:01:01.017617 930 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 09:01:01.017997 930 webserver.cc:460] Webserver started at http://127.0.98.62:41147/ using document root <none> and password file <none>
I20260619 09:01:01.018208 930 fs_manager.cc:359] Metadata directory not provided
I20260619 09:01:01.018250 930 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 09:01:01.018415 930 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 09:01:01.019485 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781859658134510-392-0/minicluster-data/master-0/data/instance:
uuid: "f41004daf1424fe8ba0b008250cdef55"
format_stamp: "Formatted at 2026-06-19 09:01:01 on dist-test-slave-mtdr"
server_key: "a3f7c4de72f715c757c0d15629932ba8"
server_key_iv: "f4b6a927bb154b5589d06d24718543af"
server_key_version: "encryptionkey@0"
I20260619 09:01:01.019786 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781859658134510-392-0/minicluster-data/master-0/wal/instance:
uuid: "f41004daf1424fe8ba0b008250cdef55"
format_stamp: "Formatted at 2026-06-19 09:01:01 on dist-test-slave-mtdr"
server_key: "a3f7c4de72f715c757c0d15629932ba8"
server_key_iv: "f4b6a927bb154b5589d06d24718543af"
server_key_version: "encryptionkey@0"
I20260619 09:01:01.021010 930 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 09:01:01.021674 945 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 09:01:01.021811 930 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 09:01:01.021888 930 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781859658134510-392-0/minicluster-data/master-0/data,/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781859658134510-392-0/minicluster-data/master-0/wal
uuid: "f41004daf1424fe8ba0b008250cdef55"
format_stamp: "Formatted at 2026-06-19 09:01:01 on dist-test-slave-mtdr"
server_key: "a3f7c4de72f715c757c0d15629932ba8"
server_key_iv: "f4b6a927bb154b5589d06d24718543af"
server_key_version: "encryptionkey@0"
I20260619 09:01:01.021951 930 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781859658134510-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781859658134510-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781859658134510-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 09:01:01.032510 930 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 09:01:01.034399 930 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 09:01:01.034508 930 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 09:01:01.038266 930 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37287
I20260619 09:01:01.038303 997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37287 every 8 connection(s)
I20260619 09:01:01.038683 930 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781859658134510-392-0/minicluster-data/master-0/data/info.pb
I20260619 09:01:01.039716 998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 09:01:01.041643 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ez5NB/build/release/bin/kudu as pid 930
I20260619 09:01:01.041726 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781859658134510-392-0/minicluster-data/master-0/wal/instance
I20260619 09:01:01.041872 392 external_mini_cluster.cc:1428] Setting key 89ddeef458dd3fed7deafb7c03b90182
I20260619 09:01:01.042171 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f41004daf1424fe8ba0b008250cdef55: Bootstrap starting.
I20260619 09:01:01.042742 998 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f41004daf1424fe8ba0b008250cdef55: Neither blocks nor log segments found. Creating new log.
I20260619 09:01:01.043035 998 log.cc:824] T 00000000000000000000000000000000 P f41004daf1424fe8ba0b008250cdef55: Log is configured to *not* fsync() on all Append() calls
I20260619 09:01:01.043654 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f41004daf1424fe8ba0b008250cdef55: No bootstrap required, opened a new log
I20260619 09:01:01.044910 998 raft_consensus.cc:348] T 00000000000000000000000000000000 P f41004daf1424fe8ba0b008250cdef55 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f41004daf1424fe8ba0b008250cdef55" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37287 } }
I20260619 09:01:01.045034 998 raft_consensus.cc:374] T 00000000000000000000000000000000 P f41004daf1424fe8ba0b008250cdef55 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 09:01:01.045058 998 raft_consensus.cc:729] T 00000000000000000000000000000000 P f41004daf1424fe8ba0b008250cdef55 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f41004daf1424fe8ba0b008250cdef55, State: Initialized, Role: FOLLOWER
I20260619 09:01:01.045177 998 consensus_queue.cc:260] T 00000000000000000000000000000000 P f41004daf1424fe8ba0b008250cdef55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f41004daf1424fe8ba0b008250cdef55" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37287 } }
I20260619 09:01:01.045238 998 raft_consensus.cc:388] T 00000000000000000000000000000000 P f41004daf1424fe8ba0b008250cdef55 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 09:01:01.045259 998 raft_consensus.cc:482] T 00000000000000000000000000000000 P f41004daf1424fe8ba0b008250cdef55 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 09:01:01.045302 998 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f41004daf1424fe8ba0b008250cdef55 [term 0 FOLLOWER]: Advancing to term 1
I20260619 09:01:01.045763 998 raft_consensus.cc:504] T 00000000000000000000000000000000 P f41004daf1424fe8ba0b008250cdef55 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f41004daf1424fe8ba0b008250cdef55" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37287 } }
I20260619 09:01:01.045850 998 leader_election.cc:302] T 00000000000000000000000000000000 P f41004daf1424fe8ba0b008250cdef55 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f41004daf1424fe8ba0b008250cdef55; no voters:
I20260619 09:01:01.045989 998 leader_election.cc:288] T 00000000000000000000000000000000 P f41004daf1424fe8ba0b008250cdef55 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 09:01:01.046022 1003 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f41004daf1424fe8ba0b008250cdef55 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 09:01:01.046144 1003 raft_consensus.cc:686] T 00000000000000000000000000000000 P f41004daf1424fe8ba0b008250cdef55 [term 1 LEADER]: Becoming Leader. State: Replica: f41004daf1424fe8ba0b008250cdef55, State: Running, Role: LEADER
I20260619 09:01:01.046233 998 sys_catalog.cc:564] T 00000000000000000000000000000000 P f41004daf1424fe8ba0b008250cdef55 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 09:01:01.046218 1003 consensus_queue.cc:237] T 00000000000000000000000000000000 P f41004daf1424fe8ba0b008250cdef55 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f41004daf1424fe8ba0b008250cdef55" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37287 } }
I20260619 09:01:01.046519 1003 sys_catalog.cc:455] T 00000000000000000000000000000000 P f41004daf1424fe8ba0b008250cdef55 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f41004daf1424fe8ba0b008250cdef55. Latest consensus state: current_term: 1 leader_uuid: "f41004daf1424fe8ba0b008250cdef55" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f41004daf1424fe8ba0b008250cdef55" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37287 } } }
I20260619 09:01:01.046587 1003 sys_catalog.cc:458] T 00000000000000000000000000000000 P f41004daf1424fe8ba0b008250cdef55 [sys.catalog]: This master's current role is: LEADER
I20260619 09:01:01.046656 1005 sys_catalog.cc:455] T 00000000000000000000000000000000 P f41004daf1424fe8ba0b008250cdef55 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f41004daf1424fe8ba0b008250cdef55" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f41004daf1424fe8ba0b008250cdef55" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37287 } } }
I20260619 09:01:01.046736 1005 sys_catalog.cc:458] T 00000000000000000000000000000000 P f41004daf1424fe8ba0b008250cdef55 [sys.catalog]: This master's current role is: LEADER
I20260619 09:01:01.046802 1008 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 09:01:01.047266 1008 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 09:01:01.048851 1008 catalog_manager.cc:1347] Generated new cluster ID: a474eecd102b4223891fb48415c283f1
I20260619 09:01:01.048900 1008 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 19 09:01:01 dist-test-slave-mtdr krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859660, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 09:01:01.053064 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:01.042332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59558 (local address 127.0.98.62:37287)
0619 09:01:01.042493 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:01.042500 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:01.042524 (+ 24us) server_negotiation.cc:407] Connection header received
0619 09:01:01.042996 (+ 472us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:01.043003 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:01.043051 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:01.043175 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:01:01.043851 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:01.044723 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:01.045735 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:01.046326 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:01.046425 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:01.050220 (+ 3795us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:01:01.050224 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:01:01.050224 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:01:01.050233 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:01:01.051697 (+ 1464us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:01:01.051993 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:01:01.051996 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:01:01.051996 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:01:01.052073 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:01:01.052234 (+ 161us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:01:01.052235 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:01:01.052235 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:01:01.052444 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:01:01.052613 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0619 09:01:01.052682 (+ 69us) server_negotiation.cc:299] Negotiation successful
0619 09:01:01.052802 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":73,"threads_started":1}
I20260619 09:01:01.061863 1008 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 09:01:01.062358 1008 catalog_manager.cc:1504] Loading token signing keys...
I20260619 09:01:01.066459 1008 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f41004daf1424fe8ba0b008250cdef55: Generated new TSK 0
I20260619 09:01:01.066640 1008 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781859658134510-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781859658134510-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 09:01:01.094983 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ez5NB/build/release/bin/kudu
/tmp/dist-test-task4ez5NB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781859658134510-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781859658134510-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781859658134510-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781859658134510-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37287
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781859658134510-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37075
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781859658134510-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781859658134510-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781859658134510-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 09:01:01.172895 1024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 09:01:01.173106 1024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 09:01:01.173141 1024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 09:01:01.174305 1024 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 09:01:01.174345 1024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 09:01:01.174381 1024 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 09:01:01.175846 1024 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37075
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781859658134510-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781859658134510-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781859658134510-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781859658134510-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37287
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1024
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781859658134510-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 09:01:01.176082 1024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 09:01:01.176308 1024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 09:01:01.180909 1029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:01:01.180966 1032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:01:01.180955 1030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:01:01.181144 1024 server_base.cc:1034] running on GCE node
I20260619 09:01:01.181316 1024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 09:01:01.181569 1024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 09:01:01.182709 1024 hybrid_clock.cc:648] HybridClock initialized: now 1781859661182690 us; error 29 us; skew 500 ppm
Jun 19 09:01:01 dist-test-slave-mtdr krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859661, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 09:01:01.185321 1024 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 09:01:01.185761 1024 webserver.cc:460] Webserver started at http://127.0.98.1:42413/ using document root <none> and password file <none>
I20260619 09:01:01.185963 1024 fs_manager.cc:359] Metadata directory not provided
I20260619 09:01:01.186010 1024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 09:01:01.186127 1024 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 09:01:01.187202 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781859658134510-392-0/minicluster-data/ts-0/data/instance:
uuid: "8a871d1a2b424c51a5f6da08a20a8d34"
format_stamp: "Formatted at 2026-06-19 09:01:01 on dist-test-slave-mtdr"
server_key: "72db75819a7998b5617c38dea08699b9"
server_key_iv: "f03acb51533119b4f084264b895bc69d"
server_key_version: "encryptionkey@0"
I20260619 09:01:01.187512 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781859658134510-392-0/minicluster-data/ts-0/wal/instance:
uuid: "8a871d1a2b424c51a5f6da08a20a8d34"
format_stamp: "Formatted at 2026-06-19 09:01:01 on dist-test-slave-mtdr"
server_key: "72db75819a7998b5617c38dea08699b9"
server_key_iv: "f03acb51533119b4f084264b895bc69d"
server_key_version: "encryptionkey@0"
I20260619 09:01:01.188733 1024 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 09:01:01.189395 1039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 09:01:01.189549 1024 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 09:01:01.189626 1024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781859658134510-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781859658134510-392-0/minicluster-data/ts-0/wal
uuid: "8a871d1a2b424c51a5f6da08a20a8d34"
format_stamp: "Formatted at 2026-06-19 09:01:01 on dist-test-slave-mtdr"
server_key: "72db75819a7998b5617c38dea08699b9"
server_key_iv: "f03acb51533119b4f084264b895bc69d"
server_key_version: "encryptionkey@0"
I20260619 09:01:01.189699 1024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781859658134510-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781859658134510-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781859658134510-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 09:01:01.208979 1024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 09:01:01.211103 1024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 09:01:01.211239 1024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 09:01:01.211457 1024 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 09:01:01.211746 1024 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 09:01:01.211779 1024 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 09:01:01.211808 1024 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 09:01:01.211825 1024 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 09:01:01.217235 1024 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34731
I20260619 09:01:01.217283 1152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34731 every 8 connection(s)
I20260619 09:01:01.217562 1024 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781859658134510-392-0/minicluster-data/ts-0/data/info.pb
I20260619 09:01:01.219570 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ez5NB/build/release/bin/kudu as pid 1024
I20260619 09:01:01.219651 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781859658134510-392-0/minicluster-data/ts-0/wal/instance
I20260619 09:01:01.219800 392 external_mini_cluster.cc:1428] Setting key 58f15fabb053b29f4b5612f48aacb393
Jun 19 09:01:01 dist-test-slave-mtdr krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859661, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 09:01:01.226745 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:01.218239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59707 (local address 127.0.98.62:37287)
0619 09:01:01.218324 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:01.218326 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:01.218510 (+ 184us) server_negotiation.cc:407] Connection header received
0619 09:01:01.219003 (+ 493us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:01.219004 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:01.219014 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:01.219062 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:01:01.219986 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:01.220567 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:01.221651 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:01.221859 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:01.221913 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:01.224213 (+ 2300us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:01:01.224215 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:01:01.224215 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:01:01.224223 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:01:01.225483 (+ 1260us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:01:01.225860 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:01:01.225861 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:01:01.225862 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:01:01.225912 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:01:01.226120 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:01:01.226121 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:01:01.226121 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:01:01.226247 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:01:01.226301 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0619 09:01:01.226562 (+ 261us) server_negotiation.cc:299] Negotiation successful
0619 09:01:01.226649 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260619 09:01:01.226954 1155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:01.218218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37287 (local address 127.0.98.1:59707)
0619 09:01:01.218397 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0619 09:01:01.218433 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0619 09:01:01.218929 (+ 496us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:01:01.219118 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:01:01.219121 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:01:01.219656 (+ 535us) client_negotiation.cc:189] Negotiated authn=SASL
0619 09:01:01.219896 (+ 240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:01.219902 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:01.220647 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:01.220648 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:01.221569 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:01.221571 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:01.221931 (+ 360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:01.221932 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:01.222068 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:01.222585 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 09:01:01.222596 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 09:01:01.224121 (+ 1525us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 09:01:01.225558 (+ 1437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:01:01.225565 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:01:01.225567 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:01:01.225795 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:01:01.225966 (+ 171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:01:01.225966 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:01:01.225967 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:01:01.226066 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:01:01.226302 (+ 236us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 09:01:01.226309 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 09:01:01.226448 (+ 139us) client_negotiation.cc:769] Sending connection context
0619 09:01:01.226537 (+ 89us) client_negotiation.cc:240] Negotiation successful
0619 09:01:01.226709 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":103,"threads_started":1}
I20260619 09:01:01.227401 1153 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37287
I20260619 09:01:01.227487 1153 heartbeater.cc:464] Registering TS with master...
I20260619 09:01:01.227651 1153 heartbeater.cc:511] Master 127.0.98.62:37287 requested a full tablet report, sending...
I20260619 09:01:01.228116 962 ts_manager.cc:194] Registered new tserver with Master: 8a871d1a2b424c51a5f6da08a20a8d34 (127.0.98.1:34731)
I20260619 09:01:01.228677 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:59707
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781859658134510-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781859658134510-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 09:01:01.241878 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ez5NB/build/release/bin/kudu
/tmp/dist-test-task4ez5NB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781859658134510-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781859658134510-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781859658134510-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781859658134510-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37287
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781859658134510-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37075
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781859658134510-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781859658134510-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781859658134510-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 09:01:01.318310 1158 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 09:01:01.318498 1158 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 09:01:01.318519 1158 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 09:01:01.319619 1158 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 09:01:01.319650 1158 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 09:01:01.319675 1158 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 09:01:01.320969 1158 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37075
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781859658134510-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781859658134510-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781859658134510-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781859658134510-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37287
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1158
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781859658134510-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 09:01:01.321152 1158 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 09:01:01.321326 1158 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 09:01:01.325946 1163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:01:01.325984 1166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:01:01.326089 1158 server_base.cc:1034] running on GCE node
W20260619 09:01:01.326192 1164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:01:01.326375 1158 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 09:01:01.326593 1158 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 09:01:01.327730 1158 hybrid_clock.cc:648] HybridClock initialized: now 1781859661327719 us; error 30 us; skew 500 ppm
Jun 19 09:01:01 dist-test-slave-mtdr krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859661, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 09:01:01.329931 1158 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 09:01:01.330338 1158 webserver.cc:460] Webserver started at http://127.0.98.2:40959/ using document root <none> and password file <none>
I20260619 09:01:01.330529 1158 fs_manager.cc:359] Metadata directory not provided
I20260619 09:01:01.330574 1158 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 09:01:01.330693 1158 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 09:01:01.331737 1158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781859658134510-392-0/minicluster-data/ts-1/data/instance:
uuid: "1ba2800f8db141e8ba5e9c3b3a431fed"
format_stamp: "Formatted at 2026-06-19 09:01:01 on dist-test-slave-mtdr"
server_key: "97c0d38ca4a62116aca779dae45a0f08"
server_key_iv: "914db8a1a22943d98bf8662296d2a5b6"
server_key_version: "encryptionkey@0"
I20260619 09:01:01.332050 1158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781859658134510-392-0/minicluster-data/ts-1/wal/instance:
uuid: "1ba2800f8db141e8ba5e9c3b3a431fed"
format_stamp: "Formatted at 2026-06-19 09:01:01 on dist-test-slave-mtdr"
server_key: "97c0d38ca4a62116aca779dae45a0f08"
server_key_iv: "914db8a1a22943d98bf8662296d2a5b6"
server_key_version: "encryptionkey@0"
I20260619 09:01:01.333226 1158 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 09:01:01.333859 1173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 09:01:01.334004 1158 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 09:01:01.334079 1158 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781859658134510-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781859658134510-392-0/minicluster-data/ts-1/wal
uuid: "1ba2800f8db141e8ba5e9c3b3a431fed"
format_stamp: "Formatted at 2026-06-19 09:01:01 on dist-test-slave-mtdr"
server_key: "97c0d38ca4a62116aca779dae45a0f08"
server_key_iv: "914db8a1a22943d98bf8662296d2a5b6"
server_key_version: "encryptionkey@0"
I20260619 09:01:01.334146 1158 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781859658134510-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781859658134510-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781859658134510-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 09:01:01.341310 1158 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 09:01:01.343104 1158 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 09:01:01.343221 1158 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 09:01:01.343415 1158 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 09:01:01.343669 1158 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 09:01:01.343700 1158 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 09:01:01.343725 1158 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 09:01:01.343750 1158 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 09:01:01.349073 1158 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40349
I20260619 09:01:01.349129 1286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40349 every 8 connection(s)
I20260619 09:01:01.349398 1158 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781859658134510-392-0/minicluster-data/ts-1/data/info.pb
Jun 19 09:01:01 dist-test-slave-mtdr krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859661, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 09:01:01.356144 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ez5NB/build/release/bin/kudu as pid 1158
I20260619 09:01:01.356225 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781859658134510-392-0/minicluster-data/ts-1/wal/instance
I20260619 09:01:01.356392 392 external_mini_cluster.cc:1428] Setting key bdeaf9a68e8c0b3c868d53f0ce702522
I20260619 09:01:01.358059 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:01.350034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53899 (local address 127.0.98.62:37287)
0619 09:01:01.350127 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:01.350129 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:01.350285 (+ 156us) server_negotiation.cc:407] Connection header received
0619 09:01:01.350745 (+ 460us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:01.350746 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:01.350757 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:01.350836 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:01:01.351713 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:01.352225 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:01.353163 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:01.353352 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:01.353415 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:01.355445 (+ 2030us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:01:01.355446 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:01:01.355446 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:01:01.355452 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:01:01.356832 (+ 1380us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:01:01.357219 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:01:01.357221 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:01:01.357221 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:01:01.357278 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:01:01.357487 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:01:01.357488 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:01:01.357489 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:01:01.357593 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:01:01.357646 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0619 09:01:01.357899 (+ 253us) server_negotiation.cc:299] Negotiation successful
0619 09:01:01.357972 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 09:01:01.358258 1289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:01.350015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37287 (local address 127.0.98.2:53899)
0619 09:01:01.350176 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0619 09:01:01.350209 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0619 09:01:01.350646 (+ 437us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:01:01.350913 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:01:01.350917 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:01:01.351444 (+ 527us) client_negotiation.cc:189] Negotiated authn=SASL
0619 09:01:01.351627 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:01.351632 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:01.352303 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:01.352304 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:01.353079 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:01.353080 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:01.353445 (+ 365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:01.353446 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:01.353561 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:01.354026 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 09:01:01.354035 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0619 09:01:01.355371 (+ 1336us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 09:01:01.356898 (+ 1527us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:01:01.356904 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:01:01.356906 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:01:01.357158 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:01:01.357330 (+ 172us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:01:01.357331 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:01:01.357331 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:01:01.357433 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:01:01.357647 (+ 214us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 09:01:01.357653 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 09:01:01.357783 (+ 130us) client_negotiation.cc:769] Sending connection context
0619 09:01:01.357872 (+ 89us) client_negotiation.cc:240] Negotiation successful
0619 09:01:01.358019 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":84,"threads_started":1}
I20260619 09:01:01.358704 1287 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37287
I20260619 09:01:01.358801 1287 heartbeater.cc:464] Registering TS with master...
I20260619 09:01:01.358999 1287 heartbeater.cc:511] Master 127.0.98.62:37287 requested a full tablet report, sending...
I20260619 09:01:01.359344 962 ts_manager.cc:194] Registered new tserver with Master: 1ba2800f8db141e8ba5e9c3b3a431fed (127.0.98.2:40349)
I20260619 09:01:01.359786 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:53899
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781859658134510-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781859658134510-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 09:01:01.377482 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ez5NB/build/release/bin/kudu
/tmp/dist-test-task4ez5NB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781859658134510-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781859658134510-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781859658134510-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781859658134510-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37287
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781859658134510-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37075
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781859658134510-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781859658134510-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781859658134510-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 09:01:01.453576 1292 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 09:01:01.453763 1292 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 09:01:01.453783 1292 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 09:01:01.454905 1292 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 09:01:01.454942 1292 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 09:01:01.454967 1292 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 09:01:01.456321 1292 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37075
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781859658134510-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781859658134510-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781859658134510-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781859658134510-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37287
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1292
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781859658134510-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 09:01:01.456557 1292 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 09:01:01.456760 1292 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 09:01:01.461323 1300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:01:01.461315 1297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:01:01.461351 1298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:01:01.461506 1292 server_base.cc:1034] running on GCE node
I20260619 09:01:01.461699 1292 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 09:01:01.461921 1292 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 09:01:01.463056 1292 hybrid_clock.cc:648] HybridClock initialized: now 1781859661463042 us; error 26 us; skew 500 ppm
Jun 19 09:01:01 dist-test-slave-mtdr krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859661, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 09:01:01.465372 1292 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 09:01:01.465736 1292 webserver.cc:460] Webserver started at http://127.0.98.3:33137/ using document root <none> and password file <none>
I20260619 09:01:01.465950 1292 fs_manager.cc:359] Metadata directory not provided
I20260619 09:01:01.465992 1292 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 09:01:01.466101 1292 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 09:01:01.467111 1292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781859658134510-392-0/minicluster-data/ts-2/data/instance:
uuid: "0dfba22e7ed449b0b7e28c2353ba2b7b"
format_stamp: "Formatted at 2026-06-19 09:01:01 on dist-test-slave-mtdr"
server_key: "5212eced056164a3766bfeeb1fe968cd"
server_key_iv: "7133a631236cc2534066dcbf34e7ffe8"
server_key_version: "encryptionkey@0"
I20260619 09:01:01.467409 1292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781859658134510-392-0/minicluster-data/ts-2/wal/instance:
uuid: "0dfba22e7ed449b0b7e28c2353ba2b7b"
format_stamp: "Formatted at 2026-06-19 09:01:01 on dist-test-slave-mtdr"
server_key: "5212eced056164a3766bfeeb1fe968cd"
server_key_iv: "7133a631236cc2534066dcbf34e7ffe8"
server_key_version: "encryptionkey@0"
I20260619 09:01:01.468572 1292 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 09:01:01.469221 1307 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 09:01:01.469380 1292 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 09:01:01.469450 1292 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781859658134510-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781859658134510-392-0/minicluster-data/ts-2/wal
uuid: "0dfba22e7ed449b0b7e28c2353ba2b7b"
format_stamp: "Formatted at 2026-06-19 09:01:01 on dist-test-slave-mtdr"
server_key: "5212eced056164a3766bfeeb1fe968cd"
server_key_iv: "7133a631236cc2534066dcbf34e7ffe8"
server_key_version: "encryptionkey@0"
I20260619 09:01:01.469508 1292 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781859658134510-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781859658134510-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781859658134510-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 09:01:01.487905 1292 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 09:01:01.489717 1292 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 09:01:01.489827 1292 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 09:01:01.490038 1292 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 09:01:01.490315 1292 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 09:01:01.490347 1292 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 09:01:01.490375 1292 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 09:01:01.490392 1292 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 09:01:01.495532 1292 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41891
I20260619 09:01:01.495601 1420 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41891 every 8 connection(s)
I20260619 09:01:01.495857 1292 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781859658134510-392-0/minicluster-data/ts-2/data/info.pb
Jun 19 09:01:01 dist-test-slave-mtdr krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859661, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 09:01:01.501633 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ez5NB/build/release/bin/kudu as pid 1292
I20260619 09:01:01.501729 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781859658134510-392-0/minicluster-data/ts-2/wal/instance
I20260619 09:01:01.501878 392 external_mini_cluster.cc:1428] Setting key 7838c6c72f4b4e895c41d4c135c342e7
I20260619 09:01:01.504469 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:01.496549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47609 (local address 127.0.98.62:37287)
0619 09:01:01.496597 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:01.496599 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:01.496777 (+ 178us) server_negotiation.cc:407] Connection header received
0619 09:01:01.497282 (+ 505us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:01.497283 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:01.497295 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:01.497347 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:01:01.498289 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:01.498784 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:01.499750 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:01.499964 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:01.500018 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:01.502015 (+ 1997us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:01:01.502016 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:01:01.502017 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:01:01.502023 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:01:01.503339 (+ 1316us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:01:01.503697 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:01:01.503698 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:01:01.503699 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:01:01.503767 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:01:01.503962 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:01:01.503963 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:01:01.503963 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:01:01.504062 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:01:01.504109 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0619 09:01:01.504340 (+ 231us) server_negotiation.cc:299] Negotiation successful
0619 09:01:01.504405 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260619 09:01:01.504557 1423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:01.496492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37287 (local address 127.0.98.3:47609)
0619 09:01:01.496649 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0619 09:01:01.496696 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0619 09:01:01.497183 (+ 487us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:01:01.497429 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:01:01.497433 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:01:01.498001 (+ 568us) client_negotiation.cc:189] Negotiated authn=SASL
0619 09:01:01.498187 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:01.498192 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:01.498888 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:01.498889 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:01.499672 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:01.499674 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:01.500031 (+ 357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:01.500032 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:01.500149 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:01.500599 (+ 450us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 09:01:01.500607 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 09:01:01.501921 (+ 1314us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 09:01:01.503412 (+ 1491us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:01:01.503417 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:01:01.503419 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:01:01.503617 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:01:01.503821 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:01:01.503822 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:01:01.503822 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:01:01.503915 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:01:01.504100 (+ 185us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 09:01:01.504102 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 09:01:01.504209 (+ 107us) client_negotiation.cc:769] Sending connection context
0619 09:01:01.504281 (+ 72us) client_negotiation.cc:240] Negotiation successful
0619 09:01:01.504401 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":74,"threads_started":1}
I20260619 09:01:01.505002 1421 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37287
I20260619 09:01:01.505088 1421 heartbeater.cc:464] Registering TS with master...
I20260619 09:01:01.505259 1421 heartbeater.cc:511] Master 127.0.98.62:37287 requested a full tablet report, sending...
I20260619 09:01:01.505569 962 ts_manager.cc:194] Registered new tserver with Master: 0dfba22e7ed449b0b7e28c2353ba2b7b (127.0.98.3:41891)
I20260619 09:01:01.505914 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:47609
I20260619 09:01:01.513510 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 09:01:01.520601 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:01.515052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59562 (local address 127.0.98.62:37287)
0619 09:01:01.515111 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:01.515113 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:01.515182 (+ 69us) server_negotiation.cc:407] Connection header received
0619 09:01:01.515244 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:01.515245 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:01.515255 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:01.515296 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:01:01.515836 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:01.516318 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:01.517005 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:01.517188 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:01.517239 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:01.518370 (+ 1131us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:01:01.518371 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:01:01.518371 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:01:01.518379 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:01:01.519581 (+ 1202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:01:01.519880 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:01:01.519882 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:01:01.519882 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:01:01.519936 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:01:01.520089 (+ 153us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:01:01.520089 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:01:01.520090 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:01:01.520233 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:01:01.520321 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0619 09:01:01.520431 (+ 110us) server_negotiation.cc:299] Negotiation successful
0619 09:01:01.520508 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260619 09:01:01.526629 962 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59562:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 09:01:01.528203 962 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 09:01:01.534435 1432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:01.530503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41891 (local address 127.0.0.1:45796)
0619 09:01:01.530919 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0619 09:01:01.530930 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 09:01:01.530989 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:01:01.531281 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:01:01.531282 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:01:01.531284 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 09:01:01.531329 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:01.531330 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:01.532579 (+ 1249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:01.532580 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:01.533375 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:01.533377 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:01.534090 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:01.534091 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:01.534177 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:01.534193 (+ 16us) client_negotiation.cc:769] Sending connection context
0619 09:01:01.534270 (+ 77us) client_negotiation.cc:240] Negotiation successful
0619 09:01:01.534332 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":400,"mutex_wait_us":7,"thread_start_us":87,"threads_started":1}
I20260619 09:01:01.534493 1431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:01.530468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32782 (local address 127.0.98.2:40349)
0619 09:01:01.530773 (+ 305us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:01.530776 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:01.530786 (+ 10us) server_negotiation.cc:407] Connection header received
0619 09:01:01.530851 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:01.530854 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:01.530870 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:01.531120 (+ 250us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 09:01:01.531195 (+ 75us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:01.532040 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:01.533458 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:01.534161 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:01.534235 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:01.534291 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0619 09:01:01.534374 (+ 83us) server_negotiation.cc:299] Negotiation successful
0619 09:01:01.534415 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":276,"thread_start_us":106,"threads_started":1}
I20260619 09:01:01.534498 1430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:01.530318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40349 (local address 127.0.0.1:32782)
0619 09:01:01.530530 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0619 09:01:01.530575 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 09:01:01.530642 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:01:01.530976 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:01:01.530979 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:01:01.530981 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 09:01:01.531047 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:01.531049 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:01.532130 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:01.532131 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:01.533368 (+ 1237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:01.533371 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:01.534234 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:01.534235 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:01.534288 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:01.534291 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 09:01:01.534357 (+ 66us) client_negotiation.cc:240] Negotiation successful
0619 09:01:01.534392 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":52,"threads_started":1}
I20260619 09:01:01.534715 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:01.530728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45796 (local address 127.0.98.3:41891)
0619 09:01:01.530933 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:01.530937 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:01.530977 (+ 40us) server_negotiation.cc:407] Connection header received
0619 09:01:01.531068 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:01.531070 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:01.531086 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:01.531166 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 09:01:01.531425 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:01.532483 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:01.533469 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:01.534004 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:01.534080 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:01.534142 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0619 09:01:01.534573 (+ 431us) server_negotiation.cc:299] Negotiation successful
0619 09:01:01.534623 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":100,"threads_started":1}
I20260619 09:01:01.534866 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:01.530522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34554 (local address 127.0.98.1:34731)
0619 09:01:01.530702 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:01.530707 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:01.530795 (+ 88us) server_negotiation.cc:407] Connection header received
0619 09:01:01.530857 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:01.530859 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:01.530871 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:01.530946 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 09:01:01.531159 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:01.532343 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:01.533589 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:01.534260 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:01.534318 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:01.534365 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0619 09:01:01.534715 (+ 350us) server_negotiation.cc:299] Negotiation successful
0619 09:01:01.534756 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":78,"threads_started":1}
I20260619 09:01:01.535087 1429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:01.530323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34731 (local address 127.0.0.1:34554)
0619 09:01:01.530673 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0619 09:01:01.530684 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 09:01:01.530743 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:01:01.531032 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:01:01.531033 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:01:01.531035 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 09:01:01.531070 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:01.531071 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:01.532465 (+ 1394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:01.532466 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:01.533510 (+ 1044us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:01.533511 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:01.534397 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:01.534398 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:01.534454 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:01.534458 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 09:01:01.534973 (+ 515us) client_negotiation.cc:240] Negotiation successful
0619 09:01:01.535022 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":74,"threads_started":1}
I20260619 09:01:01.535401 1221 tablet_service.cc:1467] Processing CreateTablet for tablet 5a1a287706c744b596e0735298d43281 (DEFAULT_TABLE table=test-table [id=1d5a3c3f6eab455883a50ccd495c2497]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 09:01:01.535441 1355 tablet_service.cc:1467] Processing CreateTablet for tablet 5a1a287706c744b596e0735298d43281 (DEFAULT_TABLE table=test-table [id=1d5a3c3f6eab455883a50ccd495c2497]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 09:01:01.535710 1221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a1a287706c744b596e0735298d43281. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 09:01:01.535808 1355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a1a287706c744b596e0735298d43281. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 09:01:01.535830 1087 tablet_service.cc:1467] Processing CreateTablet for tablet 5a1a287706c744b596e0735298d43281 (DEFAULT_TABLE table=test-table [id=1d5a3c3f6eab455883a50ccd495c2497]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 09:01:01.536123 1087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a1a287706c744b596e0735298d43281. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 09:01:01.538666 1435 tablet_bootstrap.cc:492] T 5a1a287706c744b596e0735298d43281 P 1ba2800f8db141e8ba5e9c3b3a431fed: Bootstrap starting.
I20260619 09:01:01.539088 1436 tablet_bootstrap.cc:492] T 5a1a287706c744b596e0735298d43281 P 0dfba22e7ed449b0b7e28c2353ba2b7b: Bootstrap starting.
I20260619 09:01:01.539327 1435 tablet_bootstrap.cc:654] T 5a1a287706c744b596e0735298d43281 P 1ba2800f8db141e8ba5e9c3b3a431fed: Neither blocks nor log segments found. Creating new log.
I20260619 09:01:01.539690 1437 tablet_bootstrap.cc:492] T 5a1a287706c744b596e0735298d43281 P 8a871d1a2b424c51a5f6da08a20a8d34: Bootstrap starting.
I20260619 09:01:01.539744 1436 tablet_bootstrap.cc:654] T 5a1a287706c744b596e0735298d43281 P 0dfba22e7ed449b0b7e28c2353ba2b7b: Neither blocks nor log segments found. Creating new log.
I20260619 09:01:01.539884 1435 log.cc:824] T 5a1a287706c744b596e0735298d43281 P 1ba2800f8db141e8ba5e9c3b3a431fed: Log is configured to *not* fsync() on all Append() calls
I20260619 09:01:01.540194 1436 log.cc:824] T 5a1a287706c744b596e0735298d43281 P 0dfba22e7ed449b0b7e28c2353ba2b7b: Log is configured to *not* fsync() on all Append() calls
I20260619 09:01:01.540315 1437 tablet_bootstrap.cc:654] T 5a1a287706c744b596e0735298d43281 P 8a871d1a2b424c51a5f6da08a20a8d34: Neither blocks nor log segments found. Creating new log.
I20260619 09:01:01.540638 1435 tablet_bootstrap.cc:492] T 5a1a287706c744b596e0735298d43281 P 1ba2800f8db141e8ba5e9c3b3a431fed: No bootstrap required, opened a new log
I20260619 09:01:01.540697 1435 ts_tablet_manager.cc:1397] T 5a1a287706c744b596e0735298d43281 P 1ba2800f8db141e8ba5e9c3b3a431fed: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260619 09:01:01.540778 1437 log.cc:824] T 5a1a287706c744b596e0735298d43281 P 8a871d1a2b424c51a5f6da08a20a8d34: Log is configured to *not* fsync() on all Append() calls
I20260619 09:01:01.541430 1437 tablet_bootstrap.cc:492] T 5a1a287706c744b596e0735298d43281 P 8a871d1a2b424c51a5f6da08a20a8d34: No bootstrap required, opened a new log
I20260619 09:01:01.541491 1437 ts_tablet_manager.cc:1397] T 5a1a287706c744b596e0735298d43281 P 8a871d1a2b424c51a5f6da08a20a8d34: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 09:01:01.542052 1435 raft_consensus.cc:348] T 5a1a287706c744b596e0735298d43281 P 1ba2800f8db141e8ba5e9c3b3a431fed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a871d1a2b424c51a5f6da08a20a8d34" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34731 } } peers { permanent_uuid: "0dfba22e7ed449b0b7e28c2353ba2b7b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41891 } } peers { permanent_uuid: "1ba2800f8db141e8ba5e9c3b3a431fed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40349 } }
I20260619 09:01:01.542209 1435 raft_consensus.cc:374] T 5a1a287706c744b596e0735298d43281 P 1ba2800f8db141e8ba5e9c3b3a431fed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 09:01:01.542243 1435 raft_consensus.cc:729] T 5a1a287706c744b596e0735298d43281 P 1ba2800f8db141e8ba5e9c3b3a431fed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ba2800f8db141e8ba5e9c3b3a431fed, State: Initialized, Role: FOLLOWER
I20260619 09:01:01.542349 1435 consensus_queue.cc:260] T 5a1a287706c744b596e0735298d43281 P 1ba2800f8db141e8ba5e9c3b3a431fed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a871d1a2b424c51a5f6da08a20a8d34" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34731 } } peers { permanent_uuid: "0dfba22e7ed449b0b7e28c2353ba2b7b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41891 } } peers { permanent_uuid: "1ba2800f8db141e8ba5e9c3b3a431fed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40349 } }
I20260619 09:01:01.542610 1435 ts_tablet_manager.cc:1428] T 5a1a287706c744b596e0735298d43281 P 1ba2800f8db141e8ba5e9c3b3a431fed: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 09:01:01.542629 1287 heartbeater.cc:503] Master 127.0.98.62:37287 was elected leader, sending a full tablet report...
I20260619 09:01:01.543272 1436 tablet_bootstrap.cc:492] T 5a1a287706c744b596e0735298d43281 P 0dfba22e7ed449b0b7e28c2353ba2b7b: No bootstrap required, opened a new log
I20260619 09:01:01.543380 1436 ts_tablet_manager.cc:1397] T 5a1a287706c744b596e0735298d43281 P 0dfba22e7ed449b0b7e28c2353ba2b7b: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260619 09:01:01.543905 1437 raft_consensus.cc:348] T 5a1a287706c744b596e0735298d43281 P 8a871d1a2b424c51a5f6da08a20a8d34 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a871d1a2b424c51a5f6da08a20a8d34" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34731 } } peers { permanent_uuid: "0dfba22e7ed449b0b7e28c2353ba2b7b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41891 } } peers { permanent_uuid: "1ba2800f8db141e8ba5e9c3b3a431fed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40349 } }
I20260619 09:01:01.544083 1437 raft_consensus.cc:374] T 5a1a287706c744b596e0735298d43281 P 8a871d1a2b424c51a5f6da08a20a8d34 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 09:01:01.544127 1437 raft_consensus.cc:729] T 5a1a287706c744b596e0735298d43281 P 8a871d1a2b424c51a5f6da08a20a8d34 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a871d1a2b424c51a5f6da08a20a8d34, State: Initialized, Role: FOLLOWER
I20260619 09:01:01.544245 1437 consensus_queue.cc:260] T 5a1a287706c744b596e0735298d43281 P 8a871d1a2b424c51a5f6da08a20a8d34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a871d1a2b424c51a5f6da08a20a8d34" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34731 } } peers { permanent_uuid: "0dfba22e7ed449b0b7e28c2353ba2b7b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41891 } } peers { permanent_uuid: "1ba2800f8db141e8ba5e9c3b3a431fed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40349 } }
I20260619 09:01:01.544523 1153 heartbeater.cc:503] Master 127.0.98.62:37287 was elected leader, sending a full tablet report...
I20260619 09:01:01.544523 1437 ts_tablet_manager.cc:1428] T 5a1a287706c744b596e0735298d43281 P 8a871d1a2b424c51a5f6da08a20a8d34: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260619 09:01:01.545266 1436 raft_consensus.cc:348] T 5a1a287706c744b596e0735298d43281 P 0dfba22e7ed449b0b7e28c2353ba2b7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a871d1a2b424c51a5f6da08a20a8d34" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34731 } } peers { permanent_uuid: "0dfba22e7ed449b0b7e28c2353ba2b7b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41891 } } peers { permanent_uuid: "1ba2800f8db141e8ba5e9c3b3a431fed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40349 } }
I20260619 09:01:01.545399 1436 raft_consensus.cc:374] T 5a1a287706c744b596e0735298d43281 P 0dfba22e7ed449b0b7e28c2353ba2b7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 09:01:01.545430 1436 raft_consensus.cc:729] T 5a1a287706c744b596e0735298d43281 P 0dfba22e7ed449b0b7e28c2353ba2b7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0dfba22e7ed449b0b7e28c2353ba2b7b, State: Initialized, Role: FOLLOWER
I20260619 09:01:01.545513 1436 consensus_queue.cc:260] T 5a1a287706c744b596e0735298d43281 P 0dfba22e7ed449b0b7e28c2353ba2b7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a871d1a2b424c51a5f6da08a20a8d34" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34731 } } peers { permanent_uuid: "0dfba22e7ed449b0b7e28c2353ba2b7b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41891 } } peers { permanent_uuid: "1ba2800f8db141e8ba5e9c3b3a431fed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40349 } }
I20260619 09:01:01.545801 1436 ts_tablet_manager.cc:1428] T 5a1a287706c744b596e0735298d43281 P 0dfba22e7ed449b0b7e28c2353ba2b7b: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260619 09:01:01.545838 1421 heartbeater.cc:503] Master 127.0.98.62:37287 was elected leader, sending a full tablet report...
W20260619 09:01:01.599925 1288 tablet.cc:2369] T 5a1a287706c744b596e0735298d43281 P 1ba2800f8db141e8ba5e9c3b3a431fed: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 09:01:01.672219 1443 raft_consensus.cc:482] T 5a1a287706c744b596e0735298d43281 P 0dfba22e7ed449b0b7e28c2353ba2b7b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 09:01:01.672355 1443 raft_consensus.cc:504] T 5a1a287706c744b596e0735298d43281 P 0dfba22e7ed449b0b7e28c2353ba2b7b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a871d1a2b424c51a5f6da08a20a8d34" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34731 } } peers { permanent_uuid: "0dfba22e7ed449b0b7e28c2353ba2b7b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41891 } } peers { permanent_uuid: "1ba2800f8db141e8ba5e9c3b3a431fed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40349 } }
I20260619 09:01:01.672654 1443 leader_election.cc:288] T 5a1a287706c744b596e0735298d43281 P 0dfba22e7ed449b0b7e28c2353ba2b7b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a871d1a2b424c51a5f6da08a20a8d34 (127.0.98.1:34731), 1ba2800f8db141e8ba5e9c3b3a431fed (127.0.98.2:40349)
I20260619 09:01:01.676492 1423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:01.672913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40349 (local address 127.0.98.3:60837)
0619 09:01:01.673032 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0619 09:01:01.673043 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 09:01:01.673090 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:01:01.673359 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:01:01.673360 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:01:01.673362 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 09:01:01.673431 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:01.673433 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:01.674275 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:01.674276 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:01.675471 (+ 1195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:01.675473 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:01.676216 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:01.676217 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:01.676301 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:01.676306 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 09:01:01.676363 (+ 57us) client_negotiation.cc:240] Negotiation successful
0619 09:01:01.676404 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100}
I20260619 09:01:01.676827 1431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:01.673179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60837 (local address 127.0.98.2:40349)
0619 09:01:01.673220 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:01.673221 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:01.673229 (+ 8us) server_negotiation.cc:407] Connection header received
0619 09:01:01.673255 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:01.673256 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:01.673268 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:01.673311 (+ 43us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 09:01:01.673504 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:01.674183 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:01.675546 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:01.676126 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:01.676197 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:01.676208 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0619 09:01:01.676702 (+ 494us) server_negotiation.cc:299] Negotiation successful
0619 09:01:01.676751 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260619 09:01:01.677253 1241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a1a287706c744b596e0735298d43281" candidate_uuid: "0dfba22e7ed449b0b7e28c2353ba2b7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ba2800f8db141e8ba5e9c3b3a431fed" is_pre_election: true
I20260619 09:01:01.677421 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:01.673021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38225 (local address 127.0.98.1:34731)
0619 09:01:01.673067 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:01.673070 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:01.673739 (+ 669us) server_negotiation.cc:407] Connection header received
0619 09:01:01.673841 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:01.673842 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:01.673855 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:01.673942 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 09:01:01.674178 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:01.675037 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:01.676364 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:01.677021 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:01.677100 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:01.677113 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0619 09:01:01.677311 (+ 198us) server_negotiation.cc:299] Negotiation successful
0619 09:01:01.677352 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260619 09:01:01.677435 1241 raft_consensus.cc:2454] T 5a1a287706c744b596e0735298d43281 P 1ba2800f8db141e8ba5e9c3b3a431fed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0dfba22e7ed449b0b7e28c2353ba2b7b in term 0.
I20260619 09:01:01.677618 1444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:01.672919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34731 (local address 127.0.98.3:38225)
0619 09:01:01.673509 (+ 590us) negotiation.cc:107] Waiting for socket to connect
0619 09:01:01.673685 (+ 176us) client_negotiation.cc:174] Beginning negotiation
0619 09:01:01.673742 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:01:01.673973 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:01:01.673974 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:01:01.673977 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 09:01:01.674060 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:01.674062 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:01.675120 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:01.675121 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:01.676287 (+ 1166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:01.676290 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:01.677148 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:01.677149 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:01.677222 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:01.677227 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 09:01:01.677498 (+ 271us) client_negotiation.cc:240] Negotiation successful
0619 09:01:01.677542 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":547,"mutex_wait_us":161,"thread_start_us":91,"threads_started":1}
I20260619 09:01:01.677681 1308 leader_election.cc:302] T 5a1a287706c744b596e0735298d43281 P 0dfba22e7ed449b0b7e28c2353ba2b7b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0dfba22e7ed449b0b7e28c2353ba2b7b, 1ba2800f8db141e8ba5e9c3b3a431fed; no voters:
I20260619 09:01:01.677822 1443 raft_consensus.cc:2781] T 5a1a287706c744b596e0735298d43281 P 0dfba22e7ed449b0b7e28c2353ba2b7b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 09:01:01.677901 1443 raft_consensus.cc:482] T 5a1a287706c744b596e0735298d43281 P 0dfba22e7ed449b0b7e28c2353ba2b7b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 09:01:01.677928 1443 raft_consensus.cc:3037] T 5a1a287706c744b596e0735298d43281 P 0dfba22e7ed449b0b7e28c2353ba2b7b [term 0 FOLLOWER]: Advancing to term 1
I20260619 09:01:01.677989 1107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a1a287706c744b596e0735298d43281" candidate_uuid: "0dfba22e7ed449b0b7e28c2353ba2b7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a871d1a2b424c51a5f6da08a20a8d34" is_pre_election: true
I20260619 09:01:01.678114 1107 raft_consensus.cc:2454] T 5a1a287706c744b596e0735298d43281 P 8a871d1a2b424c51a5f6da08a20a8d34 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0dfba22e7ed449b0b7e28c2353ba2b7b in term 0.
I20260619 09:01:01.678784 1443 raft_consensus.cc:504] T 5a1a287706c744b596e0735298d43281 P 0dfba22e7ed449b0b7e28c2353ba2b7b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a871d1a2b424c51a5f6da08a20a8d34" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34731 } } peers { permanent_uuid: "0dfba22e7ed449b0b7e28c2353ba2b7b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41891 } } peers { permanent_uuid: "1ba2800f8db141e8ba5e9c3b3a431fed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40349 } }
I20260619 09:01:01.678956 1443 leader_election.cc:288] T 5a1a287706c744b596e0735298d43281 P 0dfba22e7ed449b0b7e28c2353ba2b7b [CANDIDATE]: Term 1 election: Requested vote from peers 8a871d1a2b424c51a5f6da08a20a8d34 (127.0.98.1:34731), 1ba2800f8db141e8ba5e9c3b3a431fed (127.0.98.2:40349)
I20260619 09:01:01.679101 1107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a1a287706c744b596e0735298d43281" candidate_uuid: "0dfba22e7ed449b0b7e28c2353ba2b7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a871d1a2b424c51a5f6da08a20a8d34"
I20260619 09:01:01.679174 1107 raft_consensus.cc:3037] T 5a1a287706c744b596e0735298d43281 P 8a871d1a2b424c51a5f6da08a20a8d34 [term 0 FOLLOWER]: Advancing to term 1
I20260619 09:01:01.679255 1241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a1a287706c744b596e0735298d43281" candidate_uuid: "0dfba22e7ed449b0b7e28c2353ba2b7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ba2800f8db141e8ba5e9c3b3a431fed"
I20260619 09:01:01.679347 1241 raft_consensus.cc:3037] T 5a1a287706c744b596e0735298d43281 P 1ba2800f8db141e8ba5e9c3b3a431fed [term 0 FOLLOWER]: Advancing to term 1
I20260619 09:01:01.680011 1107 raft_consensus.cc:2454] T 5a1a287706c744b596e0735298d43281 P 8a871d1a2b424c51a5f6da08a20a8d34 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0dfba22e7ed449b0b7e28c2353ba2b7b in term 1.
I20260619 09:01:01.680254 1241 raft_consensus.cc:2454] T 5a1a287706c744b596e0735298d43281 P 1ba2800f8db141e8ba5e9c3b3a431fed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0dfba22e7ed449b0b7e28c2353ba2b7b in term 1.
I20260619 09:01:01.680367 1311 leader_election.cc:302] T 5a1a287706c744b596e0735298d43281 P 0dfba22e7ed449b0b7e28c2353ba2b7b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0dfba22e7ed449b0b7e28c2353ba2b7b, 8a871d1a2b424c51a5f6da08a20a8d34; no voters:
I20260619 09:01:01.680452 1443 raft_consensus.cc:2781] T 5a1a287706c744b596e0735298d43281 P 0dfba22e7ed449b0b7e28c2353ba2b7b [term 1 FOLLOWER]: Leader election won for term 1
I20260619 09:01:01.680598 1443 raft_consensus.cc:686] T 5a1a287706c744b596e0735298d43281 P 0dfba22e7ed449b0b7e28c2353ba2b7b [term 1 LEADER]: Becoming Leader. State: Replica: 0dfba22e7ed449b0b7e28c2353ba2b7b, State: Running, Role: LEADER
I20260619 09:01:01.680682 1443 consensus_queue.cc:237] T 5a1a287706c744b596e0735298d43281 P 0dfba22e7ed449b0b7e28c2353ba2b7b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a871d1a2b424c51a5f6da08a20a8d34" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34731 } } peers { permanent_uuid: "0dfba22e7ed449b0b7e28c2353ba2b7b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41891 } } peers { permanent_uuid: "1ba2800f8db141e8ba5e9c3b3a431fed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40349 } }
I20260619 09:01:01.681488 962 catalog_manager.cc:5495] T 5a1a287706c744b596e0735298d43281 P 0dfba22e7ed449b0b7e28c2353ba2b7b reported cstate change: term changed from 0 to 1, leader changed from <none> to 0dfba22e7ed449b0b7e28c2353ba2b7b (127.0.98.3). New cstate: current_term: 1 leader_uuid: "0dfba22e7ed449b0b7e28c2353ba2b7b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a871d1a2b424c51a5f6da08a20a8d34" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34731 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0dfba22e7ed449b0b7e28c2353ba2b7b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41891 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1ba2800f8db141e8ba5e9c3b3a431fed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40349 } health_report { overall_health: UNKNOWN } } }
W20260619 09:01:01.718219 1154 tablet.cc:2369] T 5a1a287706c744b596e0735298d43281 P 8a871d1a2b424c51a5f6da08a20a8d34: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 19 09:01:01 dist-test-slave-mtdr krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859660, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260619 09:01:01.723273 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:01.715424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34562 (local address 127.0.98.1:34731)
0619 09:01:01.715482 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:01.715484 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:01.715615 (+ 131us) server_negotiation.cc:407] Connection header received
0619 09:01:01.715691 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:01.715692 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:01.715703 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:01.715748 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:01:01.716287 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:01.717027 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:01.718023 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:01.718294 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:01.718364 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:01.720741 (+ 2377us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:01:01.720751 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:01:01.720751 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:01:01.720759 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:01:01.722074 (+ 1315us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:01:01.722405 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:01:01.722409 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:01:01.722409 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:01:01.722478 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:01:01.722636 (+ 158us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:01:01.722636 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:01:01.722637 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:01:01.722797 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:01:01.723000 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0619 09:01:01.723123 (+ 123us) server_negotiation.cc:299] Negotiation successful
0619 09:01:01.723192 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
Jun 19 09:01:01 dist-test-slave-mtdr krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859661, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 09:01:01 dist-test-slave-mtdr krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859661, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260619 09:01:01.743741 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:01.737557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44100 (local address 127.0.98.1:34731)
0619 09:01:01.737597 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:01.737600 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:01.737661 (+ 61us) server_negotiation.cc:407] Connection header received
0619 09:01:01.737729 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:01.737730 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:01.737740 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:01.737785 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:01:01.738282 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:01.738743 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:01.739472 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:01.739650 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:01.739702 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:01.741685 (+ 1983us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:01:01.741686 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:01:01.741687 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:01:01.741694 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:01:01.742693 (+ 999us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:01:01.743027 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:01:01.743028 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:01:01.743029 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:01:01.743096 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:01:01.743256 (+ 160us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:01:01.743256 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:01:01.743257 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:01:01.743343 (+ 86us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:01:01.743424 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0619 09:01:01.743575 (+ 151us) server_negotiation.cc:299] Negotiation successful
0619 09:01:01.743651 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
W20260619 09:01:01.746408 1422 tablet.cc:2369] T 5a1a287706c744b596e0735298d43281 P 0dfba22e7ed449b0b7e28c2353ba2b7b: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 19 09:01:01 dist-test-slave-mtdr krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859661, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 09:01:01 dist-test-slave-mtdr krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859661, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260619 09:01:01.763756 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:01.757407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44112 (local address 127.0.98.1:34731)
0619 09:01:01.757456 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:01.757458 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:01.757567 (+ 109us) server_negotiation.cc:407] Connection header received
0619 09:01:01.757628 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:01.757629 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:01.757640 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:01.757683 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:01:01.758156 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:01.758606 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:01.759331 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:01.759506 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:01.759559 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:01.761567 (+ 2008us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:01:01.761568 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:01:01.761569 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:01:01.761576 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:01:01.762754 (+ 1178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:01:01.763107 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:01:01.763108 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:01:01.763109 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:01:01.763171 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:01:01.763333 (+ 162us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:01:01.763334 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:01:01.763334 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:01:01.763421 (+ 87us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:01:01.763502 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0619 09:01:01.763597 (+ 95us) server_negotiation.cc:299] Negotiation successful
0619 09:01:01.763666 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
W20260619 09:01:01.764183 1067 tablet_service.cc:2908] Rejecting scan request for tablet 5a1a287706c744b596e0735298d43281: Uninitialized: clean time has not yet been initialized
I20260619 09:01:01.764931 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ez5NB/build/release/bin/kudu with pid 1024
I20260619 09:01:01.768849 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ez5NB/build/release/bin/kudu with pid 1158
I20260619 09:01:01.772445 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ez5NB/build/release/bin/kudu with pid 1292
I20260619 09:01:01.776170 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ez5NB/build/release/bin/kudu with pid 930
2026-06-19T09:01:01Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (1907 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781859658134510-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 09:01:01 dist-test-slave-mtdr krb5kdc[1459](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 09:01:01 dist-test-slave-mtdr krb5kdc[1459](info): set up 2 sockets
Jun 19 09:01:01 dist-test-slave-mtdr krb5kdc[1459](info): commencing operation
krb5kdc: starting...
W20260619 09:01:02.741077 392 mini_kdc.cc:121] Time spent starting KDC: real 0.946s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
W20260619 09:01:03.021916 392 mini_kdc.cc:253] Time spent creating user principal test-admin: real 0.281s user 0.001s sys 0.001s
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
W20260619 09:01:03.232888 392 mini_kdc.cc:253] Time spent creating user principal test-user: real 0.211s user 0.000s sys 0.002s
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
W20260619 09:01:03.482328 392 mini_kdc.cc:253] Time spent creating user principal joe-interloper: real 0.249s user 0.000s sys 0.001s
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781859658134510-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781859658134510-392-0/krb5kdc/test-user.keytab.
W20260619 09:01:04.534538 392 mini_kdc.cc:302] Time spent creating keytab for test-user: real 1.052s user 0.000s sys 0.001s
Jun 19 09:01:04 dist-test-slave-mtdr krb5kdc[1459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859664, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T09:01:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T09:01:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781859658134510-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781859658134510-392-0/krb5kdc/kudu_127.0.98.62.keytab.
W20260619 09:01:05.166543 392 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.0.98.62: real 0.613s user 0.000s sys 0.002s
I20260619 09:01:05.166906 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ez5NB/build/release/bin/kudu
/tmp/dist-test-task4ez5NB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781859658134510-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781859658134510-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781859658134510-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781859658134510-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43583
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781859658134510-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40479
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43583
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781859658134510-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781859658134510-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781859658134510-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 09:01:05.246596 1472 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 09:01:05.246789 1472 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 09:01:05.246809 1472 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 09:01:05.247962 1472 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 09:01:05.248000 1472 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260619 09:01:05.248013 1472 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 09:01:05.248025 1472 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 09:01:05.248035 1472 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 09:01:05.248055 1472 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 09:01:05.249522 1472 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40479
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781859658134510-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781859658134510-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43583
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781859658134510-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43583
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781859658134510-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1472
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781859658134510-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 09:01:05.249732 1472 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 09:01:05.249941 1472 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 09:01:05.254536 1477 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:01:05.254596 1478 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:01:05.254612 1472 server_base.cc:1034] running on GCE node
W20260619 09:01:05.254566 1480 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:01:05.255077 1472 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 09:01:05.255369 1472 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 09:01:05.256501 1472 hybrid_clock.cc:648] HybridClock initialized: now 1781859665256483 us; error 35 us; skew 500 ppm
Jun 19 09:01:05 dist-test-slave-mtdr krb5kdc[1459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859665, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 09:01:05.261962 1472 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 09:01:05.262377 1472 webserver.cc:460] Webserver started at http://127.0.98.62:34151/ using document root <none> and password file <none>
I20260619 09:01:05.262584 1472 fs_manager.cc:359] Metadata directory not provided
I20260619 09:01:05.262629 1472 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 09:01:05.262718 1472 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 09:01:05.263783 1472 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781859658134510-392-0/minicluster-data/master-0/data/instance:
uuid: "30c28e2314774e6089287721c79bb73a"
format_stamp: "Formatted at 2026-06-19 09:01:05 on dist-test-slave-mtdr"
server_key: "24c124842b61bc351a6f94df8c273e44"
server_key_iv: "14ef8235d67cdecf94314c02f26b4b60"
server_key_version: "encryptionkey@0"
I20260619 09:01:05.264068 1472 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781859658134510-392-0/minicluster-data/master-0/wal/instance:
uuid: "30c28e2314774e6089287721c79bb73a"
format_stamp: "Formatted at 2026-06-19 09:01:05 on dist-test-slave-mtdr"
server_key: "24c124842b61bc351a6f94df8c273e44"
server_key_iv: "14ef8235d67cdecf94314c02f26b4b60"
server_key_version: "encryptionkey@0"
I20260619 09:01:05.265223 1472 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 09:01:05.265818 1487 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 09:01:05.265973 1472 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 09:01:05.266076 1472 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781859658134510-392-0/minicluster-data/master-0/data,/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781859658134510-392-0/minicluster-data/master-0/wal
uuid: "30c28e2314774e6089287721c79bb73a"
format_stamp: "Formatted at 2026-06-19 09:01:05 on dist-test-slave-mtdr"
server_key: "24c124842b61bc351a6f94df8c273e44"
server_key_iv: "14ef8235d67cdecf94314c02f26b4b60"
server_key_version: "encryptionkey@0"
I20260619 09:01:05.266130 1472 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781859658134510-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781859658134510-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781859658134510-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 09:01:05.292950 1472 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 09:01:05.295171 1472 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 09:01:05.295275 1472 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 09:01:05.299381 1472 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43583
I20260619 09:01:05.299458 1549 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43583 every 8 connection(s)
I20260619 09:01:05.299733 1472 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781859658134510-392-0/minicluster-data/master-0/data/info.pb
I20260619 09:01:05.300352 1550 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 09:01:05.301529 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ez5NB/build/release/bin/kudu as pid 1472
I20260619 09:01:05.301609 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781859658134510-392-0/minicluster-data/master-0/wal/instance
I20260619 09:01:05.301757 392 external_mini_cluster.cc:1428] Setting key 0eeb0eae014b961f3045bef5a60d146e
I20260619 09:01:05.302920 1550 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 30c28e2314774e6089287721c79bb73a: Bootstrap starting.
I20260619 09:01:05.303474 1550 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 30c28e2314774e6089287721c79bb73a: Neither blocks nor log segments found. Creating new log.
I20260619 09:01:05.303730 1550 log.cc:824] T 00000000000000000000000000000000 P 30c28e2314774e6089287721c79bb73a: Log is configured to *not* fsync() on all Append() calls
I20260619 09:01:05.304328 1550 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 30c28e2314774e6089287721c79bb73a: No bootstrap required, opened a new log
I20260619 09:01:05.306026 1550 raft_consensus.cc:348] T 00000000000000000000000000000000 P 30c28e2314774e6089287721c79bb73a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30c28e2314774e6089287721c79bb73a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43583 } }
I20260619 09:01:05.306164 1550 raft_consensus.cc:374] T 00000000000000000000000000000000 P 30c28e2314774e6089287721c79bb73a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 09:01:05.306196 1550 raft_consensus.cc:729] T 00000000000000000000000000000000 P 30c28e2314774e6089287721c79bb73a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30c28e2314774e6089287721c79bb73a, State: Initialized, Role: FOLLOWER
I20260619 09:01:05.306317 1550 consensus_queue.cc:260] T 00000000000000000000000000000000 P 30c28e2314774e6089287721c79bb73a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30c28e2314774e6089287721c79bb73a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43583 } }
I20260619 09:01:05.306391 1550 raft_consensus.cc:388] T 00000000000000000000000000000000 P 30c28e2314774e6089287721c79bb73a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 09:01:05.306425 1550 raft_consensus.cc:482] T 00000000000000000000000000000000 P 30c28e2314774e6089287721c79bb73a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 09:01:05.306469 1550 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 30c28e2314774e6089287721c79bb73a [term 0 FOLLOWER]: Advancing to term 1
Jun 19 09:01:05 dist-test-slave-mtdr krb5kdc[1459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859664, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 09:01:05.310894 1552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:05.302247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50510 (local address 127.0.98.62:43583)
0619 09:01:05.302393 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:05.302400 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:05.302641 (+ 241us) server_negotiation.cc:407] Connection header received
0619 09:01:05.303114 (+ 473us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:05.303121 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:05.303172 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:05.303296 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:01:05.303897 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:05.304804 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:05.305492 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:05.305911 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:05.306002 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:05.308238 (+ 2236us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:01:05.308241 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:01:05.308241 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:01:05.308251 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:01:05.309596 (+ 1345us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:01:05.309886 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:01:05.309889 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:01:05.309889 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:01:05.309961 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:01:05.310129 (+ 168us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:01:05.310130 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:01:05.310130 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:01:05.310338 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:01:05.310461 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0619 09:01:05.310588 (+ 127us) server_negotiation.cc:299] Negotiation successful
0619 09:01:05.310711 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":65,"threads_started":1}
I20260619 09:01:05.333901 1550 raft_consensus.cc:504] T 00000000000000000000000000000000 P 30c28e2314774e6089287721c79bb73a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30c28e2314774e6089287721c79bb73a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43583 } }
I20260619 09:01:05.334069 1550 leader_election.cc:302] T 00000000000000000000000000000000 P 30c28e2314774e6089287721c79bb73a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 30c28e2314774e6089287721c79bb73a; no voters:
I20260619 09:01:05.334301 1550 leader_election.cc:288] T 00000000000000000000000000000000 P 30c28e2314774e6089287721c79bb73a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 09:01:05.334352 1555 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 30c28e2314774e6089287721c79bb73a [term 1 FOLLOWER]: Leader election won for term 1
I20260619 09:01:05.334559 1550 sys_catalog.cc:564] T 00000000000000000000000000000000 P 30c28e2314774e6089287721c79bb73a [sys.catalog]: configured and running, proceeding with master startup.
I20260619 09:01:05.334578 1555 raft_consensus.cc:686] T 00000000000000000000000000000000 P 30c28e2314774e6089287721c79bb73a [term 1 LEADER]: Becoming Leader. State: Replica: 30c28e2314774e6089287721c79bb73a, State: Running, Role: LEADER
I20260619 09:01:05.334738 1555 consensus_queue.cc:237] T 00000000000000000000000000000000 P 30c28e2314774e6089287721c79bb73a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30c28e2314774e6089287721c79bb73a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43583 } }
I20260619 09:01:05.335167 1556 sys_catalog.cc:455] T 00000000000000000000000000000000 P 30c28e2314774e6089287721c79bb73a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "30c28e2314774e6089287721c79bb73a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30c28e2314774e6089287721c79bb73a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43583 } } }
I20260619 09:01:05.335247 1556 sys_catalog.cc:458] T 00000000000000000000000000000000 P 30c28e2314774e6089287721c79bb73a [sys.catalog]: This master's current role is: LEADER
I20260619 09:01:05.335376 1557 sys_catalog.cc:455] T 00000000000000000000000000000000 P 30c28e2314774e6089287721c79bb73a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 30c28e2314774e6089287721c79bb73a. Latest consensus state: current_term: 1 leader_uuid: "30c28e2314774e6089287721c79bb73a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30c28e2314774e6089287721c79bb73a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43583 } } }
I20260619 09:01:05.335431 1557 sys_catalog.cc:458] T 00000000000000000000000000000000 P 30c28e2314774e6089287721c79bb73a [sys.catalog]: This master's current role is: LEADER
W20260619 09:01:05.336359 1570 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 30c28e2314774e6089287721c79bb73a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 09:01:05.336400 1570 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 09:01:05.336438 1564 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 09:01:05.336576 1564 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 09:01:05.337901 1564 catalog_manager.cc:1347] Generated new cluster ID: 6805019b1ba64d8aa05edb22f21c53be
I20260619 09:01:05.337947 1564 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 09:01:05.343106 1564 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 09:01:05.343575 1564 catalog_manager.cc:1504] Loading token signing keys...
I20260619 09:01:05.354107 1564 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 30c28e2314774e6089287721c79bb73a: Generated new TSK 0
I20260619 09:01:05.354270 1564 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781859658134510-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781859658134510-392-0/krb5kdc/kudu_127.0.98.1.keytab.
W20260619 09:01:06.144820 392 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.0.98.1: real 0.786s user 0.000s sys 0.003s
I20260619 09:01:06.145140 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ez5NB/build/release/bin/kudu
/tmp/dist-test-task4ez5NB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781859658134510-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781859658134510-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781859658134510-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781859658134510-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43583
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781859658134510-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40479
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781859658134510-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781859658134510-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781859658134510-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 09:01:06.222913 1576 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 09:01:06.223115 1576 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 09:01:06.223147 1576 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 09:01:06.224330 1576 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 09:01:06.224373 1576 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 09:01:06.224401 1576 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 09:01:06.224423 1576 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 09:01:06.225900 1576 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40479
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781859658134510-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781859658134510-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781859658134510-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781859658134510-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:43583
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1576
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781859658134510-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 09:01:06.226137 1576 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 09:01:06.226346 1576 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 09:01:06.231036 1584 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:01:06.231040 1582 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:01:06.231074 1581 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:01:06.231303 1576 server_base.cc:1034] running on GCE node
I20260619 09:01:06.231487 1576 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 09:01:06.231693 1576 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 09:01:06.232834 1576 hybrid_clock.cc:648] HybridClock initialized: now 1781859666232823 us; error 30 us; skew 500 ppm
Jun 19 09:01:06 dist-test-slave-mtdr krb5kdc[1459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859666, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 09:01:06.235287 1576 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 09:01:06.235705 1576 webserver.cc:460] Webserver started at http://127.0.98.1:42111/ using document root <none> and password file <none>
I20260619 09:01:06.235896 1576 fs_manager.cc:359] Metadata directory not provided
I20260619 09:01:06.235953 1576 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 09:01:06.236078 1576 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 09:01:06.237069 1576 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781859658134510-392-0/minicluster-data/ts-0/data/instance:
uuid: "374b600775294768a44620bdbaa906e7"
format_stamp: "Formatted at 2026-06-19 09:01:06 on dist-test-slave-mtdr"
server_key: "7548162fab4899efa8f48b69671a33ed"
server_key_iv: "2f7e8f28902c75c25221141b50985254"
server_key_version: "encryptionkey@0"
I20260619 09:01:06.237397 1576 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781859658134510-392-0/minicluster-data/ts-0/wal/instance:
uuid: "374b600775294768a44620bdbaa906e7"
format_stamp: "Formatted at 2026-06-19 09:01:06 on dist-test-slave-mtdr"
server_key: "7548162fab4899efa8f48b69671a33ed"
server_key_iv: "2f7e8f28902c75c25221141b50985254"
server_key_version: "encryptionkey@0"
I20260619 09:01:06.238600 1576 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260619 09:01:06.239365 1591 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 09:01:06.239512 1576 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260619 09:01:06.239593 1576 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781859658134510-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781859658134510-392-0/minicluster-data/ts-0/wal
uuid: "374b600775294768a44620bdbaa906e7"
format_stamp: "Formatted at 2026-06-19 09:01:06 on dist-test-slave-mtdr"
server_key: "7548162fab4899efa8f48b69671a33ed"
server_key_iv: "2f7e8f28902c75c25221141b50985254"
server_key_version: "encryptionkey@0"
I20260619 09:01:06.239657 1576 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781859658134510-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781859658134510-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781859658134510-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 09:01:06.252714 1576 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 09:01:06.254621 1576 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 09:01:06.254750 1576 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 09:01:06.255265 1576 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 09:01:06.255304 1576 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 09:01:06.255331 1576 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 09:01:06.255348 1576 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 09:01:06 dist-test-slave-mtdr krb5kdc[1459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859666, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 09:01:06.261983 1576 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:32863
I20260619 09:01:06.262027 1707 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:32863 every 8 connection(s)
I20260619 09:01:06.262339 1576 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781859658134510-392-0/minicluster-data/ts-0/data/info.pb
I20260619 09:01:06.264439 1600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:06.255665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49505 (local address 127.0.98.62:43583)
0619 09:01:06.255768 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:06.255771 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:06.255934 (+ 163us) server_negotiation.cc:407] Connection header received
0619 09:01:06.256475 (+ 541us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:06.256476 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:06.256492 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:06.256573 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:01:06.257203 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:06.257798 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:06.258877 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:06.259088 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:06.259143 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:06.261640 (+ 2497us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:01:06.261641 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:01:06.261642 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:01:06.261649 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:01:06.263231 (+ 1582us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:01:06.263584 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:01:06.263585 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:01:06.263585 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:01:06.263644 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:01:06.263843 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:01:06.263844 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:01:06.263844 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:01:06.263956 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:01:06.264025 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0619 09:01:06.264266 (+ 241us) server_negotiation.cc:299] Negotiation successful
0619 09:01:06.264343 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81,"thread_start_us":50,"threads_started":1}
I20260619 09:01:06.264600 1601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:06.255660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43583 (local address 127.0.98.1:49505)
0619 09:01:06.255818 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0619 09:01:06.255858 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 09:01:06.256372 (+ 514us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:01:06.256646 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:01:06.256651 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:01:06.256917 (+ 266us) client_negotiation.cc:189] Negotiated authn=SASL
0619 09:01:06.257106 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:06.257112 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:06.257882 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:06.257883 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:06.258763 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:06.258766 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:06.259167 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:06.259169 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:06.259322 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:06.259852 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 09:01:06.259870 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 09:01:06.261513 (+ 1643us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 09:01:06.263302 (+ 1789us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:01:06.263308 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:01:06.263309 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:01:06.263508 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:01:06.263695 (+ 187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:01:06.263696 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:01:06.263696 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:01:06.263790 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:01:06.264015 (+ 225us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 09:01:06.264021 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 09:01:06.264149 (+ 128us) client_negotiation.cc:769] Sending connection context
0619 09:01:06.264240 (+ 91us) client_negotiation.cc:240] Negotiation successful
0619 09:01:06.264385 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":80,"threads_started":1}
I20260619 09:01:06.265031 1708 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43583
I20260619 09:01:06.265110 1708 heartbeater.cc:464] Registering TS with master...
I20260619 09:01:06.265270 1708 heartbeater.cc:511] Master 127.0.98.62:43583 requested a full tablet report, sending...
I20260619 09:01:06.265772 1504 ts_manager.cc:194] Registered new tserver with Master: 374b600775294768a44620bdbaa906e7 (127.0.98.1:32863)
I20260619 09:01:06.266395 1504 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:49505
I20260619 09:01:06.269621 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ez5NB/build/release/bin/kudu as pid 1576
I20260619 09:01:06.269711 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781859658134510-392-0/minicluster-data/ts-0/wal/instance
I20260619 09:01:06.269870 392 external_mini_cluster.cc:1428] Setting key 5f623c058162b3c582dea1434d3019c7
I20260619 09:01:06.271939 1600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:06.266560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34441 (local address 127.0.98.62:43583)
0619 09:01:06.266626 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:06.266627 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:06.266636 (+ 9us) server_negotiation.cc:407] Connection header received
0619 09:01:06.266654 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:06.266655 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:06.266664 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:06.266715 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:01:06.267310 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:06.267773 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:06.268458 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:06.268716 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:06.268783 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:06.269612 (+ 829us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:01:06.269613 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:01:06.269613 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:01:06.269619 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:01:06.270933 (+ 1314us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:01:06.271258 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:01:06.271259 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:01:06.271260 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:01:06.271301 (+ 41us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:01:06.271471 (+ 170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:01:06.271472 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:01:06.271472 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:01:06.271584 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:01:06.271641 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0619 09:01:06.271778 (+ 137us) server_negotiation.cc:299] Negotiation successful
0619 09:01:06.271844 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260619 09:01:06.271950 1714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:06.265849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43583 (local address 127.0.98.1:34441)
0619 09:01:06.265969 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0619 09:01:06.265978 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 09:01:06.266019 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:01:06.266735 (+ 716us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:01:06.266736 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:01:06.266951 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0619 09:01:06.267028 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:06.267030 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:06.267871 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:06.267872 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:06.268389 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:06.268390 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:06.268769 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:06.268770 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:06.268831 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:06.269220 (+ 389us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 09:01:06.269225 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 09:01:06.269492 (+ 267us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 09:01:06.271000 (+ 1508us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:01:06.271001 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:01:06.271001 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:01:06.271189 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:01:06.271350 (+ 161us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:01:06.271351 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:01:06.271352 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:01:06.271408 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:01:06.271646 (+ 238us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 09:01:06.271647 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 09:01:06.271691 (+ 44us) client_negotiation.cc:769] Sending connection context
0619 09:01:06.271751 (+ 60us) client_negotiation.cc:240] Negotiation successful
0619 09:01:06.271844 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":84,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781859658134510-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781859658134510-392-0/krb5kdc/kudu_127.0.98.2.keytab.
W20260619 09:01:06.810009 392 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.0.98.2: real 0.539s user 0.000s sys 0.003s
I20260619 09:01:06.810328 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ez5NB/build/release/bin/kudu
/tmp/dist-test-task4ez5NB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781859658134510-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781859658134510-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781859658134510-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781859658134510-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43583
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781859658134510-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40479
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781859658134510-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781859658134510-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781859658134510-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 09:01:06.890254 1717 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 09:01:06.890450 1717 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 09:01:06.890473 1717 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 09:01:06.891644 1717 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 09:01:06.891685 1717 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 09:01:06.891703 1717 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 09:01:06.891719 1717 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 09:01:06.893175 1717 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40479
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781859658134510-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781859658134510-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781859658134510-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781859658134510-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:43583
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1717
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781859658134510-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 09:01:06.893359 1717 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 09:01:06.893545 1717 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 09:01:06.898363 1725 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:01:06.898355 1723 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:01:06.898363 1722 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:01:06.898677 1717 server_base.cc:1034] running on GCE node
I20260619 09:01:06.898891 1717 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 09:01:06.899121 1717 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 09:01:06.900254 1717 hybrid_clock.cc:648] HybridClock initialized: now 1781859666900232 us; error 48 us; skew 500 ppm
Jun 19 09:01:06 dist-test-slave-mtdr krb5kdc[1459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859666, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 09:01:06.902777 1717 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 09:01:06.903240 1717 webserver.cc:460] Webserver started at http://127.0.98.2:39841/ using document root <none> and password file <none>
I20260619 09:01:06.903427 1717 fs_manager.cc:359] Metadata directory not provided
I20260619 09:01:06.903466 1717 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 09:01:06.903555 1717 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 09:01:06.904583 1717 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781859658134510-392-0/minicluster-data/ts-1/data/instance:
uuid: "7a73cdd031584af38e35564c1e20b7c8"
format_stamp: "Formatted at 2026-06-19 09:01:06 on dist-test-slave-mtdr"
server_key: "c49d87345fcbab220ebe5de830083d05"
server_key_iv: "98dac5fbdca56b9dbb45afa98fa4cf2a"
server_key_version: "encryptionkey@0"
I20260619 09:01:06.904862 1717 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781859658134510-392-0/minicluster-data/ts-1/wal/instance:
uuid: "7a73cdd031584af38e35564c1e20b7c8"
format_stamp: "Formatted at 2026-06-19 09:01:06 on dist-test-slave-mtdr"
server_key: "c49d87345fcbab220ebe5de830083d05"
server_key_iv: "98dac5fbdca56b9dbb45afa98fa4cf2a"
server_key_version: "encryptionkey@0"
I20260619 09:01:06.906046 1717 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 09:01:06.906723 1732 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 09:01:06.906913 1717 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260619 09:01:06.907002 1717 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781859658134510-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781859658134510-392-0/minicluster-data/ts-1/wal
uuid: "7a73cdd031584af38e35564c1e20b7c8"
format_stamp: "Formatted at 2026-06-19 09:01:06 on dist-test-slave-mtdr"
server_key: "c49d87345fcbab220ebe5de830083d05"
server_key_iv: "98dac5fbdca56b9dbb45afa98fa4cf2a"
server_key_version: "encryptionkey@0"
I20260619 09:01:06.907069 1717 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781859658134510-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781859658134510-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781859658134510-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 09:01:06.933643 1717 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 09:01:06.935825 1717 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 09:01:06.935961 1717 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 09:01:06.936539 1717 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 09:01:06.936581 1717 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 09:01:06.936611 1717 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 09:01:06.936628 1717 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 09:01:06 dist-test-slave-mtdr krb5kdc[1459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859666, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 09:01:06.943181 1717 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:46601
I20260619 09:01:06.943234 1848 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:46601 every 8 connection(s)
I20260619 09:01:06.943614 1717 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781859658134510-392-0/minicluster-data/ts-1/data/info.pb
I20260619 09:01:06.944988 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ez5NB/build/release/bin/kudu as pid 1717
I20260619 09:01:06.945075 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781859658134510-392-0/minicluster-data/ts-1/wal/instance
I20260619 09:01:06.945248 392 external_mini_cluster.cc:1428] Setting key eeb7ad1e75e18108249477c21a22172f
I20260619 09:01:06.945639 1742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:06.937036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:32885 (local address 127.0.98.62:43583)
0619 09:01:06.937145 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:06.937147 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:06.937291 (+ 144us) server_negotiation.cc:407] Connection header received
0619 09:01:06.937852 (+ 561us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:06.937853 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:06.937868 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:06.937948 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:01:06.938746 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:06.939516 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:06.940551 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:06.940874 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:06.940941 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:06.943020 (+ 2079us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:01:06.943021 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:01:06.943022 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:01:06.943030 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:01:06.944408 (+ 1378us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:01:06.944734 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:01:06.944735 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:01:06.944735 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:01:06.944780 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:01:06.944991 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:01:06.944992 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:01:06.944992 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:01:06.945119 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:01:06.945181 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0619 09:01:06.945475 (+ 294us) server_negotiation.cc:299] Negotiation successful
0619 09:01:06.945560 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88,"thread_start_us":56,"threads_started":1}
I20260619 09:01:06.945811 1741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:06.936997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43583 (local address 127.0.98.2:32885)
0619 09:01:06.937167 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0619 09:01:06.937208 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 09:01:06.937713 (+ 505us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:01:06.938028 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:01:06.938032 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:01:06.938375 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0619 09:01:06.938650 (+ 275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:06.938657 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:06.939624 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:06.939625 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:06.940479 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:06.940482 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:06.940972 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:06.940973 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:06.941106 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:06.941538 (+ 432us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 09:01:06.941547 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0619 09:01:06.942929 (+ 1382us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 09:01:06.944473 (+ 1544us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:01:06.944475 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:01:06.944477 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:01:06.944678 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:01:06.944816 (+ 138us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:01:06.944816 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:01:06.944817 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:01:06.944938 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:01:06.945171 (+ 233us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 09:01:06.945178 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 09:01:06.945326 (+ 148us) client_negotiation.cc:769] Sending connection context
0619 09:01:06.945432 (+ 106us) client_negotiation.cc:240] Negotiation successful
0619 09:01:06.945594 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":72,"threads_started":1}
I20260619 09:01:06.946233 1849 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43583
I20260619 09:01:06.946311 1849 heartbeater.cc:464] Registering TS with master...
I20260619 09:01:06.946520 1849 heartbeater.cc:511] Master 127.0.98.62:43583 requested a full tablet report, sending...
I20260619 09:01:06.946893 1503 ts_manager.cc:194] Registered new tserver with Master: 7a73cdd031584af38e35564c1e20b7c8 (127.0.98.2:46601)
I20260619 09:01:06.947347 1503 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:32885
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260619 09:01:06.952570 1855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:06.947137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43583 (local address 127.0.98.2:50561)
0619 09:01:06.947475 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0619 09:01:06.947485 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 09:01:06.947539 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:01:06.948024 (+ 485us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:01:06.948025 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:01:06.948173 (+ 148us) client_negotiation.cc:189] Negotiated authn=SASL
0619 09:01:06.948215 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:06.948216 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:06.948844 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:06.948845 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:06.949342 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:06.949344 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:06.949669 (+ 325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:06.949670 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:06.949730 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:06.950081 (+ 351us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 09:01:06.950094 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 09:01:06.950346 (+ 252us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 09:01:06.951691 (+ 1345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:01:06.951692 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:01:06.951693 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:01:06.951839 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:01:06.951992 (+ 153us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:01:06.951993 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:01:06.951993 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:01:06.952044 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:01:06.952280 (+ 236us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 09:01:06.952281 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 09:01:06.952318 (+ 37us) client_negotiation.cc:769] Sending connection context
0619 09:01:06.952378 (+ 60us) client_negotiation.cc:240] Negotiation successful
0619 09:01:06.952468 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":66,"threads_started":1}
I20260619 09:01:06.952610 1742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:06.947189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50561 (local address 127.0.98.62:43583)
0619 09:01:06.947422 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:06.947424 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:06.947529 (+ 105us) server_negotiation.cc:407] Connection header received
0619 09:01:06.947580 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:06.947581 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:06.947599 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:06.947653 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:01:06.948281 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:06.948753 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:06.949408 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:06.949612 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:06.949663 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:06.950420 (+ 757us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:01:06.950421 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:01:06.950421 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:01:06.950427 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:01:06.951621 (+ 1194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:01:06.951894 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:01:06.951895 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:01:06.951895 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:01:06.951942 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:01:06.952092 (+ 150us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:01:06.952093 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:01:06.952093 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:01:06.952224 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:01:06.952279 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0619 09:01:06.952454 (+ 175us) server_negotiation.cc:299] Negotiation successful
0619 09:01:06.952522 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":215}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260619 09:01:07.267303 1708 heartbeater.cc:503] Master 127.0.98.62:43583 was elected leader, sending a full tablet report...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781859658134510-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781859658134510-392-0/krb5kdc/kudu_127.0.98.3.keytab.
W20260619 09:01:07.702484 392 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.0.98.3: real 0.756s user 0.000s sys 0.003s
I20260619 09:01:07.702838 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ez5NB/build/release/bin/kudu
/tmp/dist-test-task4ez5NB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781859658134510-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781859658134510-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781859658134510-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781859658134510-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43583
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781859658134510-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40479
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781859658134510-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781859658134510-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781859658134510-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 09:01:07.782177 1858 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 09:01:07.782384 1858 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 09:01:07.782416 1858 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 09:01:07.783555 1858 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 09:01:07.783594 1858 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 09:01:07.783623 1858 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 09:01:07.783644 1858 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 09:01:07.785146 1858 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40479
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781859658134510-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781859658134510-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781859658134510-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781859658134510-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:43583
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1858
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781859658134510-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 09:01:07.785379 1858 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 09:01:07.785578 1858 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 09:01:07.790307 1863 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:01:07.790300 1866 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:01:07.790359 1858 server_base.cc:1034] running on GCE node
W20260619 09:01:07.790310 1864 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:01:07.790771 1858 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 09:01:07.791018 1858 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 09:01:07.792199 1858 hybrid_clock.cc:648] HybridClock initialized: now 1781859667792172 us; error 31 us; skew 500 ppm
Jun 19 09:01:07 dist-test-slave-mtdr krb5kdc[1459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859667, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 09:01:07.794679 1858 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 09:01:07.795133 1858 webserver.cc:460] Webserver started at http://127.0.98.3:34553/ using document root <none> and password file <none>
I20260619 09:01:07.795346 1858 fs_manager.cc:359] Metadata directory not provided
I20260619 09:01:07.795392 1858 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 09:01:07.795511 1858 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 09:01:07.796540 1858 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781859658134510-392-0/minicluster-data/ts-2/data/instance:
uuid: "176dc359022b40eb964b3e93025673ac"
format_stamp: "Formatted at 2026-06-19 09:01:07 on dist-test-slave-mtdr"
server_key: "b9b5515b0e7e8ae843d92cd844e9f60f"
server_key_iv: "d5cbf06deb240d8b70dbc9d335f30388"
server_key_version: "encryptionkey@0"
I20260619 09:01:07.796865 1858 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781859658134510-392-0/minicluster-data/ts-2/wal/instance:
uuid: "176dc359022b40eb964b3e93025673ac"
format_stamp: "Formatted at 2026-06-19 09:01:07 on dist-test-slave-mtdr"
server_key: "b9b5515b0e7e8ae843d92cd844e9f60f"
server_key_iv: "d5cbf06deb240d8b70dbc9d335f30388"
server_key_version: "encryptionkey@0"
I20260619 09:01:07.798118 1858 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 09:01:07.798902 1873 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 09:01:07.799062 1858 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260619 09:01:07.799141 1858 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781859658134510-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781859658134510-392-0/minicluster-data/ts-2/wal
uuid: "176dc359022b40eb964b3e93025673ac"
format_stamp: "Formatted at 2026-06-19 09:01:07 on dist-test-slave-mtdr"
server_key: "b9b5515b0e7e8ae843d92cd844e9f60f"
server_key_iv: "d5cbf06deb240d8b70dbc9d335f30388"
server_key_version: "encryptionkey@0"
I20260619 09:01:07.799217 1858 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781859658134510-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781859658134510-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781859658134510-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 09:01:07.807751 1858 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 09:01:07.810169 1858 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 09:01:07.810318 1858 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 09:01:07.810984 1858 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 09:01:07.811035 1858 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 09:01:07.811111 1858 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 09:01:07.811141 1858 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 09:01:07.818854 1858 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33683
Jun 19 09:01:07 dist-test-slave-mtdr krb5kdc[1459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859667, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 09:01:07.818926 1989 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33683 every 8 connection(s)
I20260619 09:01:07.819283 1858 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781859658134510-392-0/minicluster-data/ts-2/data/info.pb
I20260619 09:01:07.822290 1883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:07.811554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40167 (local address 127.0.98.62:43583)
0619 09:01:07.811653 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:07.811654 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:07.811664 (+ 10us) server_negotiation.cc:407] Connection header received
0619 09:01:07.812060 (+ 396us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:07.812062 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:07.812075 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:07.812133 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:01:07.813120 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:07.813852 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:07.815388 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:07.815684 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:07.815737 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:07.819475 (+ 3738us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:01:07.819477 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:01:07.819477 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:01:07.819486 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:01:07.820873 (+ 1387us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:01:07.821293 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:01:07.821294 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:01:07.821294 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:01:07.821356 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:01:07.821674 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:01:07.821675 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:01:07.821676 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:01:07.821817 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:01:07.821895 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0619 09:01:07.822142 (+ 247us) server_negotiation.cc:299] Negotiation successful
0619 09:01:07.822218 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82,"thread_start_us":46,"threads_started":1}
I20260619 09:01:07.822407 1882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:07.811275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43583 (local address 127.0.98.3:40167)
0619 09:01:07.811422 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0619 09:01:07.811473 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0619 09:01:07.811965 (+ 492us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:01:07.812252 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:01:07.812257 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:01:07.812701 (+ 444us) client_negotiation.cc:189] Negotiated authn=SASL
0619 09:01:07.812983 (+ 282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:07.812991 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:07.813970 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:07.813971 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:07.815263 (+ 1292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:07.815267 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:07.815855 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:07.815856 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:07.816014 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:07.816661 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 09:01:07.816713 (+ 52us) client_negotiation.cc:656] calling sasl_client_start()
0619 09:01:07.819331 (+ 2618us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 09:01:07.820964 (+ 1633us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:01:07.820970 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:01:07.820971 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:01:07.821202 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:01:07.821476 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:01:07.821477 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:01:07.821477 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:01:07.821586 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:01:07.821884 (+ 298us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 09:01:07.821887 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 09:01:07.822005 (+ 118us) client_negotiation.cc:769] Sending connection context
0619 09:01:07.822090 (+ 85us) client_negotiation.cc:240] Negotiation successful
0619 09:01:07.822226 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":72,"threads_started":1}
I20260619 09:01:07.822952 1990 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43583
I20260619 09:01:07.823037 1990 heartbeater.cc:464] Registering TS with master...
I20260619 09:01:07.823203 1990 heartbeater.cc:511] Master 127.0.98.62:43583 requested a full tablet report, sending...
I20260619 09:01:07.823570 1504 ts_manager.cc:194] Registered new tserver with Master: 176dc359022b40eb964b3e93025673ac (127.0.98.3:33683)
I20260619 09:01:07.824034 1504 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:40167
I20260619 09:01:07.827271 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ez5NB/build/release/bin/kudu as pid 1858
I20260619 09:01:07.827365 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781859658134510-392-0/minicluster-data/ts-2/wal/instance
I20260619 09:01:07.827529 392 external_mini_cluster.cc:1428] Setting key 939f7b712454a0c269f306f26ec3dc25
I20260619 09:01:07.828358 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 09:01:07.829058 1883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:07.823993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44551 (local address 127.0.98.62:43583)
0619 09:01:07.824069 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:07.824070 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:07.824133 (+ 63us) server_negotiation.cc:407] Connection header received
0619 09:01:07.824209 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:07.824209 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:07.824235 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:07.824296 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:01:07.824592 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:07.825077 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:07.825741 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:07.825936 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:07.825989 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:07.826739 (+ 750us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:01:07.826740 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:01:07.826741 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:01:07.826747 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:01:07.828104 (+ 1357us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:01:07.828444 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:01:07.828445 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:01:07.828445 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:01:07.828499 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:01:07.828656 (+ 157us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:01:07.828656 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:01:07.828656 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:01:07.828761 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:01:07.828806 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0619 09:01:07.828938 (+ 132us) server_negotiation.cc:299] Negotiation successful
0619 09:01:07.828997 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 09:01:07.829108 1996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:07.823950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43583 (local address 127.0.98.3:44551)
0619 09:01:07.824070 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0619 09:01:07.824080 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 09:01:07.824158 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:01:07.824306 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:01:07.824306 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:01:07.824471 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0619 09:01:07.824521 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:07.824522 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:07.825161 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:07.825162 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:07.825670 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:07.825671 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:07.825993 (+ 322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:07.825994 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:07.826057 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:07.826410 (+ 353us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 09:01:07.826414 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 09:01:07.826658 (+ 244us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 09:01:07.828170 (+ 1512us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:01:07.828171 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:01:07.828172 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:01:07.828372 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:01:07.828549 (+ 177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:01:07.828549 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:01:07.828550 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:01:07.828609 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:01:07.828810 (+ 201us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 09:01:07.828811 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 09:01:07.828854 (+ 43us) client_negotiation.cc:769] Sending connection context
0619 09:01:07.828918 (+ 64us) client_negotiation.cc:240] Negotiation successful
0619 09:01:07.829011 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":70,"threads_started":1}
Jun 19 09:01:07 dist-test-slave-mtdr krb5kdc[1459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859667, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 09:01:07 dist-test-slave-mtdr krb5kdc[1459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859667, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 09:01:07.849251 1883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:07.843034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50524 (local address 127.0.98.62:43583)
0619 09:01:07.843090 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:07.843092 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:07.843126 (+ 34us) server_negotiation.cc:407] Connection header received
0619 09:01:07.843212 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:07.843213 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:07.843223 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:07.843268 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:01:07.843811 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:07.844300 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:07.845002 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:07.845195 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:07.845248 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:07.847375 (+ 2127us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:01:07.847377 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:01:07.847377 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:01:07.847384 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:01:07.848430 (+ 1046us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:01:07.848734 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:01:07.848735 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:01:07.848735 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:01:07.848776 (+ 41us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:01:07.848899 (+ 123us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:01:07.848900 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:01:07.848900 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:01:07.848984 (+ 84us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:01:07.849049 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0619 09:01:07.849116 (+ 67us) server_negotiation.cc:299] Negotiation successful
0619 09:01:07.849168 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260619 09:01:07.850795 1504 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50524:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 09:01:07.852331 1504 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 09:01:07.858335 2008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:07.854421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33683 (local address 127.0.0.1:46150)
0619 09:01:07.854588 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0619 09:01:07.854625 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0619 09:01:07.854681 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:01:07.855099 (+ 418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:01:07.855100 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:01:07.855102 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 09:01:07.855148 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:07.855150 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:07.856392 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:07.856393 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:07.857420 (+ 1027us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:07.857422 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:07.858045 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:07.858046 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:07.858120 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:07.858133 (+ 13us) client_negotiation.cc:769] Sending connection context
0619 09:01:07.858203 (+ 70us) client_negotiation.cc:240] Negotiation successful
0619 09:01:07.858257 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":68,"threads_started":1}
I20260619 09:01:07.858347 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:07.854790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46150 (local address 127.0.98.3:33683)
0619 09:01:07.854975 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:07.854979 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:07.854988 (+ 9us) server_negotiation.cc:407] Connection header received
0619 09:01:07.855020 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:07.855022 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:07.855035 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:07.855100 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 09:01:07.855268 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:07.856316 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:07.857495 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:07.857974 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:07.858035 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:07.858109 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0619 09:01:07.858222 (+ 113us) server_negotiation.cc:299] Negotiation successful
0619 09:01:07.858258 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":117,"threads_started":1}
I20260619 09:01:07.858631 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:07.854976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55114 (local address 127.0.98.1:32863)
0619 09:01:07.855110 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:07.855114 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:07.855219 (+ 105us) server_negotiation.cc:407] Connection header received
0619 09:01:07.855317 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:07.855319 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:07.855335 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:07.855386 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 09:01:07.855570 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:07.856535 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:07.857538 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:07.858201 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:07.858256 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:07.858310 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0619 09:01:07.858523 (+ 213us) server_negotiation.cc:299] Negotiation successful
0619 09:01:07.858562 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":51,"threads_started":1}
I20260619 09:01:07.858486 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:07.854421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46601 (local address 127.0.0.1:38596)
0619 09:01:07.854756 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0619 09:01:07.854764 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0619 09:01:07.854838 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:01:07.855027 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:01:07.855030 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:01:07.855033 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 09:01:07.855096 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:07.855097 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:07.856174 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:07.856175 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:07.857373 (+ 1198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:07.857375 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:07.858278 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:07.858279 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:07.858337 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:07.858341 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 09:01:07.858369 (+ 28us) client_negotiation.cc:240] Negotiation successful
0619 09:01:07.858406 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":317,"mutex_wait_us":3,"thread_start_us":64,"threads_started":1}
I20260619 09:01:07.858577 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:07.854617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:32863 (local address 127.0.0.1:55114)
0619 09:01:07.855164 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0619 09:01:07.855174 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 09:01:07.855240 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:01:07.855464 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:01:07.855465 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:01:07.855467 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 09:01:07.855498 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:07.855499 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:07.856602 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:07.856603 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:07.857459 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:07.857461 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:07.858338 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:07.858339 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:07.858388 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:07.858391 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 09:01:07.858430 (+ 39us) client_negotiation.cc:240] Negotiation successful
0619 09:01:07.858461 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":521,"thread_start_us":109,"threads_started":1}
I20260619 09:01:07.858793 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:07.854501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38596 (local address 127.0.98.2:46601)
0619 09:01:07.854618 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:07.854623 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:07.854858 (+ 235us) server_negotiation.cc:407] Connection header received
0619 09:01:07.854897 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:07.854900 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:07.854917 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:07.854992 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 09:01:07.855184 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:07.856089 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:07.857458 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:07.858179 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:07.858572 (+ 393us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:07.858640 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0619 09:01:07.858664 (+ 24us) server_negotiation.cc:299] Negotiation successful
0619 09:01:07.858717 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90,"thread_start_us":69,"threads_started":1}
I20260619 09:01:07.859359 1783 tablet_service.cc:1467] Processing CreateTablet for tablet 6c324ba315d24d959c6acd106c03b042 (DEFAULT_TABLE table=test-table [id=0fec3e5848304167899560ff92c7b41b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 09:01:07.859364 1924 tablet_service.cc:1467] Processing CreateTablet for tablet 6c324ba315d24d959c6acd106c03b042 (DEFAULT_TABLE table=test-table [id=0fec3e5848304167899560ff92c7b41b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 09:01:07.859407 1642 tablet_service.cc:1467] Processing CreateTablet for tablet 6c324ba315d24d959c6acd106c03b042 (DEFAULT_TABLE table=test-table [id=0fec3e5848304167899560ff92c7b41b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 09:01:07.859710 1924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c324ba315d24d959c6acd106c03b042. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 09:01:07.859714 1783 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c324ba315d24d959c6acd106c03b042. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 09:01:07.859721 1642 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c324ba315d24d959c6acd106c03b042. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 09:01:07.861918 2014 tablet_bootstrap.cc:492] T 6c324ba315d24d959c6acd106c03b042 P 374b600775294768a44620bdbaa906e7: Bootstrap starting.
I20260619 09:01:07.862576 2014 tablet_bootstrap.cc:654] T 6c324ba315d24d959c6acd106c03b042 P 374b600775294768a44620bdbaa906e7: Neither blocks nor log segments found. Creating new log.
I20260619 09:01:07.862733 2015 tablet_bootstrap.cc:492] T 6c324ba315d24d959c6acd106c03b042 P 7a73cdd031584af38e35564c1e20b7c8: Bootstrap starting.
I20260619 09:01:07.863055 2016 tablet_bootstrap.cc:492] T 6c324ba315d24d959c6acd106c03b042 P 176dc359022b40eb964b3e93025673ac: Bootstrap starting.
I20260619 09:01:07.863197 2014 log.cc:824] T 6c324ba315d24d959c6acd106c03b042 P 374b600775294768a44620bdbaa906e7: Log is configured to *not* fsync() on all Append() calls
I20260619 09:01:07.863363 2015 tablet_bootstrap.cc:654] T 6c324ba315d24d959c6acd106c03b042 P 7a73cdd031584af38e35564c1e20b7c8: Neither blocks nor log segments found. Creating new log.
I20260619 09:01:07.863665 2016 tablet_bootstrap.cc:654] T 6c324ba315d24d959c6acd106c03b042 P 176dc359022b40eb964b3e93025673ac: Neither blocks nor log segments found. Creating new log.
I20260619 09:01:07.863667 2015 log.cc:824] T 6c324ba315d24d959c6acd106c03b042 P 7a73cdd031584af38e35564c1e20b7c8: Log is configured to *not* fsync() on all Append() calls
I20260619 09:01:07.863915 2014 tablet_bootstrap.cc:492] T 6c324ba315d24d959c6acd106c03b042 P 374b600775294768a44620bdbaa906e7: No bootstrap required, opened a new log
I20260619 09:01:07.863973 2014 ts_tablet_manager.cc:1397] T 6c324ba315d24d959c6acd106c03b042 P 374b600775294768a44620bdbaa906e7: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 09:01:07.863956 2016 log.cc:824] T 6c324ba315d24d959c6acd106c03b042 P 176dc359022b40eb964b3e93025673ac: Log is configured to *not* fsync() on all Append() calls
I20260619 09:01:07.864395 2015 tablet_bootstrap.cc:492] T 6c324ba315d24d959c6acd106c03b042 P 7a73cdd031584af38e35564c1e20b7c8: No bootstrap required, opened a new log
I20260619 09:01:07.864490 2015 ts_tablet_manager.cc:1397] T 6c324ba315d24d959c6acd106c03b042 P 7a73cdd031584af38e35564c1e20b7c8: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260619 09:01:07.864603 2016 tablet_bootstrap.cc:492] T 6c324ba315d24d959c6acd106c03b042 P 176dc359022b40eb964b3e93025673ac: No bootstrap required, opened a new log
I20260619 09:01:07.864662 2016 ts_tablet_manager.cc:1397] T 6c324ba315d24d959c6acd106c03b042 P 176dc359022b40eb964b3e93025673ac: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 09:01:07.866096 2016 raft_consensus.cc:348] T 6c324ba315d24d959c6acd106c03b042 P 176dc359022b40eb964b3e93025673ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a73cdd031584af38e35564c1e20b7c8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46601 } } peers { permanent_uuid: "176dc359022b40eb964b3e93025673ac" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33683 } } peers { permanent_uuid: "374b600775294768a44620bdbaa906e7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32863 } }
I20260619 09:01:07.866226 2016 raft_consensus.cc:374] T 6c324ba315d24d959c6acd106c03b042 P 176dc359022b40eb964b3e93025673ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 09:01:07.866256 2016 raft_consensus.cc:729] T 6c324ba315d24d959c6acd106c03b042 P 176dc359022b40eb964b3e93025673ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 176dc359022b40eb964b3e93025673ac, State: Initialized, Role: FOLLOWER
I20260619 09:01:07.866415 2016 consensus_queue.cc:260] T 6c324ba315d24d959c6acd106c03b042 P 176dc359022b40eb964b3e93025673ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a73cdd031584af38e35564c1e20b7c8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46601 } } peers { permanent_uuid: "176dc359022b40eb964b3e93025673ac" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33683 } } peers { permanent_uuid: "374b600775294768a44620bdbaa906e7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32863 } }
I20260619 09:01:07.866676 1990 heartbeater.cc:503] Master 127.0.98.62:43583 was elected leader, sending a full tablet report...
I20260619 09:01:07.866593 2014 raft_consensus.cc:348] T 6c324ba315d24d959c6acd106c03b042 P 374b600775294768a44620bdbaa906e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a73cdd031584af38e35564c1e20b7c8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46601 } } peers { permanent_uuid: "176dc359022b40eb964b3e93025673ac" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33683 } } peers { permanent_uuid: "374b600775294768a44620bdbaa906e7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32863 } }
I20260619 09:01:07.866739 2014 raft_consensus.cc:374] T 6c324ba315d24d959c6acd106c03b042 P 374b600775294768a44620bdbaa906e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 09:01:07.866832 2014 raft_consensus.cc:729] T 6c324ba315d24d959c6acd106c03b042 P 374b600775294768a44620bdbaa906e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 374b600775294768a44620bdbaa906e7, State: Initialized, Role: FOLLOWER
I20260619 09:01:07.866667 2016 ts_tablet_manager.cc:1428] T 6c324ba315d24d959c6acd106c03b042 P 176dc359022b40eb964b3e93025673ac: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 09:01:07.866951 2014 consensus_queue.cc:260] T 6c324ba315d24d959c6acd106c03b042 P 374b600775294768a44620bdbaa906e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a73cdd031584af38e35564c1e20b7c8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46601 } } peers { permanent_uuid: "176dc359022b40eb964b3e93025673ac" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33683 } } peers { permanent_uuid: "374b600775294768a44620bdbaa906e7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32863 } }
I20260619 09:01:07.867204 2014 ts_tablet_manager.cc:1428] T 6c324ba315d24d959c6acd106c03b042 P 374b600775294768a44620bdbaa906e7: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260619 09:01:07.867271 2015 raft_consensus.cc:348] T 6c324ba315d24d959c6acd106c03b042 P 7a73cdd031584af38e35564c1e20b7c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a73cdd031584af38e35564c1e20b7c8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46601 } } peers { permanent_uuid: "176dc359022b40eb964b3e93025673ac" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33683 } } peers { permanent_uuid: "374b600775294768a44620bdbaa906e7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32863 } }
I20260619 09:01:07.867455 2015 raft_consensus.cc:374] T 6c324ba315d24d959c6acd106c03b042 P 7a73cdd031584af38e35564c1e20b7c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 09:01:07.867532 2015 raft_consensus.cc:729] T 6c324ba315d24d959c6acd106c03b042 P 7a73cdd031584af38e35564c1e20b7c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a73cdd031584af38e35564c1e20b7c8, State: Initialized, Role: FOLLOWER
I20260619 09:01:07.867686 2015 consensus_queue.cc:260] T 6c324ba315d24d959c6acd106c03b042 P 7a73cdd031584af38e35564c1e20b7c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a73cdd031584af38e35564c1e20b7c8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46601 } } peers { permanent_uuid: "176dc359022b40eb964b3e93025673ac" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33683 } } peers { permanent_uuid: "374b600775294768a44620bdbaa906e7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32863 } }
I20260619 09:01:07.867928 2015 ts_tablet_manager.cc:1428] T 6c324ba315d24d959c6acd106c03b042 P 7a73cdd031584af38e35564c1e20b7c8: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260619 09:01:07.867955 1849 heartbeater.cc:503] Master 127.0.98.62:43583 was elected leader, sending a full tablet report...
I20260619 09:01:07.917990 2020 raft_consensus.cc:482] T 6c324ba315d24d959c6acd106c03b042 P 176dc359022b40eb964b3e93025673ac [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 09:01:07.918121 2020 raft_consensus.cc:504] T 6c324ba315d24d959c6acd106c03b042 P 176dc359022b40eb964b3e93025673ac [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a73cdd031584af38e35564c1e20b7c8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46601 } } peers { permanent_uuid: "176dc359022b40eb964b3e93025673ac" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33683 } } peers { permanent_uuid: "374b600775294768a44620bdbaa906e7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32863 } }
I20260619 09:01:07.918411 2020 leader_election.cc:288] T 6c324ba315d24d959c6acd106c03b042 P 176dc359022b40eb964b3e93025673ac [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a73cdd031584af38e35564c1e20b7c8 (127.0.98.2:46601), 374b600775294768a44620bdbaa906e7 (127.0.98.1:32863)
I20260619 09:01:07.922310 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:07.918702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49733 (local address 127.0.98.2:46601)
0619 09:01:07.918754 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:07.918756 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:07.918769 (+ 13us) server_negotiation.cc:407] Connection header received
0619 09:01:07.918865 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:07.918866 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:07.918878 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:07.918953 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 09:01:07.919097 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:07.919873 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:07.921365 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:07.921924 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:07.921998 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:07.922008 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0619 09:01:07.922201 (+ 193us) server_negotiation.cc:299] Negotiation successful
0619 09:01:07.922240 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260619 09:01:07.922310 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:07.918847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41117 (local address 127.0.98.1:32863)
0619 09:01:07.918889 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:07.918891 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:07.918963 (+ 72us) server_negotiation.cc:407] Connection header received
0619 09:01:07.919007 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:07.919008 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:07.919018 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:07.919057 (+ 39us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 09:01:07.919182 (+ 125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:07.919872 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:07.921366 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:07.921930 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:07.921998 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:07.922008 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0619 09:01:07.922198 (+ 190us) server_negotiation.cc:299] Negotiation successful
0619 09:01:07.922234 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260619 09:01:07.922327 1882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:07.918638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46601 (local address 127.0.98.3:49733)
0619 09:01:07.918722 (+ 84us) negotiation.cc:107] Waiting for socket to connect
0619 09:01:07.918734 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 09:01:07.918779 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:01:07.918964 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:01:07.918965 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:01:07.918966 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 09:01:07.919035 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:07.919037 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:07.919967 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:07.919968 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:07.921260 (+ 1292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:07.921263 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:07.922023 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:07.922024 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:07.922124 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:07.922126 (+ 2us) client_negotiation.cc:769] Sending connection context
0619 09:01:07.922182 (+ 56us) client_negotiation.cc:240] Negotiation successful
0619 09:01:07.922232 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33,"mutex_wait_us":69}
I20260619 09:01:07.922333 2023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:07.918742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:32863 (local address 127.0.98.3:41117)
0619 09:01:07.918905 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0619 09:01:07.918913 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0619 09:01:07.918958 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:01:07.919061 (+ 103us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:01:07.919062 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:01:07.919063 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 09:01:07.919123 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:07.919125 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:07.919967 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:07.919968 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:07.921260 (+ 1292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:07.921263 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:07.922023 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:07.922024 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:07.922116 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:07.922121 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 09:01:07.922188 (+ 67us) client_negotiation.cc:240] Negotiation successful
0619 09:01:07.922232 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"mutex_wait_us":16,"thread_start_us":118,"threads_started":1}
I20260619 09:01:07.922740 1803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c324ba315d24d959c6acd106c03b042" candidate_uuid: "176dc359022b40eb964b3e93025673ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a73cdd031584af38e35564c1e20b7c8" is_pre_election: true
I20260619 09:01:07.922927 1803 raft_consensus.cc:2454] T 6c324ba315d24d959c6acd106c03b042 P 7a73cdd031584af38e35564c1e20b7c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 176dc359022b40eb964b3e93025673ac in term 0.
I20260619 09:01:07.922924 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c324ba315d24d959c6acd106c03b042" candidate_uuid: "176dc359022b40eb964b3e93025673ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "374b600775294768a44620bdbaa906e7" is_pre_election: true
I20260619 09:01:07.923075 1662 raft_consensus.cc:2454] T 6c324ba315d24d959c6acd106c03b042 P 374b600775294768a44620bdbaa906e7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 176dc359022b40eb964b3e93025673ac in term 0.
I20260619 09:01:07.923163 1876 leader_election.cc:302] T 6c324ba315d24d959c6acd106c03b042 P 176dc359022b40eb964b3e93025673ac [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 176dc359022b40eb964b3e93025673ac, 7a73cdd031584af38e35564c1e20b7c8; no voters:
I20260619 09:01:07.923302 2020 raft_consensus.cc:2781] T 6c324ba315d24d959c6acd106c03b042 P 176dc359022b40eb964b3e93025673ac [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 09:01:07.923359 2020 raft_consensus.cc:482] T 6c324ba315d24d959c6acd106c03b042 P 176dc359022b40eb964b3e93025673ac [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 09:01:07.923374 2020 raft_consensus.cc:3037] T 6c324ba315d24d959c6acd106c03b042 P 176dc359022b40eb964b3e93025673ac [term 0 FOLLOWER]: Advancing to term 1
I20260619 09:01:07.924376 2020 raft_consensus.cc:504] T 6c324ba315d24d959c6acd106c03b042 P 176dc359022b40eb964b3e93025673ac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a73cdd031584af38e35564c1e20b7c8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46601 } } peers { permanent_uuid: "176dc359022b40eb964b3e93025673ac" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33683 } } peers { permanent_uuid: "374b600775294768a44620bdbaa906e7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32863 } }
I20260619 09:01:07.924482 2020 leader_election.cc:288] T 6c324ba315d24d959c6acd106c03b042 P 176dc359022b40eb964b3e93025673ac [CANDIDATE]: Term 1 election: Requested vote from peers 7a73cdd031584af38e35564c1e20b7c8 (127.0.98.2:46601), 374b600775294768a44620bdbaa906e7 (127.0.98.1:32863)
I20260619 09:01:07.924638 1803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c324ba315d24d959c6acd106c03b042" candidate_uuid: "176dc359022b40eb964b3e93025673ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a73cdd031584af38e35564c1e20b7c8"
I20260619 09:01:07.924669 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c324ba315d24d959c6acd106c03b042" candidate_uuid: "176dc359022b40eb964b3e93025673ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "374b600775294768a44620bdbaa906e7"
I20260619 09:01:07.924714 1803 raft_consensus.cc:3037] T 6c324ba315d24d959c6acd106c03b042 P 7a73cdd031584af38e35564c1e20b7c8 [term 0 FOLLOWER]: Advancing to term 1
I20260619 09:01:07.924732 1662 raft_consensus.cc:3037] T 6c324ba315d24d959c6acd106c03b042 P 374b600775294768a44620bdbaa906e7 [term 0 FOLLOWER]: Advancing to term 1
I20260619 09:01:07.925372 1662 raft_consensus.cc:2454] T 6c324ba315d24d959c6acd106c03b042 P 374b600775294768a44620bdbaa906e7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 176dc359022b40eb964b3e93025673ac in term 1.
I20260619 09:01:07.925387 1803 raft_consensus.cc:2454] T 6c324ba315d24d959c6acd106c03b042 P 7a73cdd031584af38e35564c1e20b7c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 176dc359022b40eb964b3e93025673ac in term 1.
I20260619 09:01:07.925524 1876 leader_election.cc:302] T 6c324ba315d24d959c6acd106c03b042 P 176dc359022b40eb964b3e93025673ac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 176dc359022b40eb964b3e93025673ac, 374b600775294768a44620bdbaa906e7; no voters:
I20260619 09:01:07.925611 2020 raft_consensus.cc:2781] T 6c324ba315d24d959c6acd106c03b042 P 176dc359022b40eb964b3e93025673ac [term 1 FOLLOWER]: Leader election won for term 1
I20260619 09:01:07.925740 2020 raft_consensus.cc:686] T 6c324ba315d24d959c6acd106c03b042 P 176dc359022b40eb964b3e93025673ac [term 1 LEADER]: Becoming Leader. State: Replica: 176dc359022b40eb964b3e93025673ac, State: Running, Role: LEADER
I20260619 09:01:07.925818 2020 consensus_queue.cc:237] T 6c324ba315d24d959c6acd106c03b042 P 176dc359022b40eb964b3e93025673ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a73cdd031584af38e35564c1e20b7c8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46601 } } peers { permanent_uuid: "176dc359022b40eb964b3e93025673ac" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33683 } } peers { permanent_uuid: "374b600775294768a44620bdbaa906e7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32863 } }
I20260619 09:01:07.926623 1504 catalog_manager.cc:5495] T 6c324ba315d24d959c6acd106c03b042 P 176dc359022b40eb964b3e93025673ac reported cstate change: term changed from 0 to 1, leader changed from <none> to 176dc359022b40eb964b3e93025673ac (127.0.98.3). New cstate: current_term: 1 leader_uuid: "176dc359022b40eb964b3e93025673ac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a73cdd031584af38e35564c1e20b7c8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "176dc359022b40eb964b3e93025673ac" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33683 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "374b600775294768a44620bdbaa906e7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32863 } health_report { overall_health: UNKNOWN } } }
W20260619 09:01:07.944598 1850 tablet.cc:2369] T 6c324ba315d24d959c6acd106c03b042 P 7a73cdd031584af38e35564c1e20b7c8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 09:01:07.949994 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:07.947859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46160 (local address 127.0.98.3:33683)
0619 09:01:07.947915 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:07.947916 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:07.947923 (+ 7us) server_negotiation.cc:407] Connection header received
0619 09:01:07.948045 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:07.948046 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:07.948066 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:07.948127 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 09:01:07.948250 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:07.948702 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:07.949393 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:07.949565 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:07.949606 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:07.949762 (+ 156us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 09:01:07.949848 (+ 86us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 09:01:07.949900 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0619 09:01:07.949926 (+ 26us) server_negotiation.cc:299] Negotiation successful
0619 09:01:07.949942 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260619 09:01:07.951623 1662 raft_consensus.cc:1261] T 6c324ba315d24d959c6acd106c03b042 P 374b600775294768a44620bdbaa906e7 [term 1 FOLLOWER]: Refusing update from remote peer 176dc359022b40eb964b3e93025673ac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 09:01:07.951622 1803 raft_consensus.cc:1261] T 6c324ba315d24d959c6acd106c03b042 P 7a73cdd031584af38e35564c1e20b7c8 [term 1 FOLLOWER]: Refusing update from remote peer 176dc359022b40eb964b3e93025673ac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 09:01:07.951941 2020 consensus_queue.cc:1035] T 6c324ba315d24d959c6acd106c03b042 P 176dc359022b40eb964b3e93025673ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a73cdd031584af38e35564c1e20b7c8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 09:01:07.952080 2024 consensus_queue.cc:1035] T 6c324ba315d24d959c6acd106c03b042 P 176dc359022b40eb964b3e93025673ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "374b600775294768a44620bdbaa906e7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 09:01:07.954208 2027 mvcc.cc:205] Tried to move back new op lower bound from 7298497199926996992 to 7298497199824429056. Current Snapshot: MvccSnapshot[applied={T|T < 7298497199926996992}]
I20260619 09:01:07.954269 2028 mvcc.cc:205] Tried to move back new op lower bound from 7298497199926996992 to 7298497199824429056. Current Snapshot: MvccSnapshot[applied={T|T < 7298497199926996992}]
I20260619 09:01:07.954278 2029 mvcc.cc:205] Tried to move back new op lower bound from 7298497199926996992 to 7298497199824429056. Current Snapshot: MvccSnapshot[applied={T|T < 7298497199926996992}]
I20260619 09:01:07.957101 1504 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50524:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 09:01:07.957183 1504 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50524:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 09:01:07.957842 1504 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 30c28e2314774e6089287721c79bb73a: Sending DeleteTablet for 3 replicas of tablet 6c324ba315d24d959c6acd106c03b042
I20260619 09:01:07.958117 1783 tablet_service.cc:1514] Processing DeleteTablet for tablet 6c324ba315d24d959c6acd106c03b042 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 09:01:07 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38596
I20260619 09:01:07.958168 1924 tablet_service.cc:1514] Processing DeleteTablet for tablet 6c324ba315d24d959c6acd106c03b042 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 09:01:07 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46150
I20260619 09:01:07.958163 1642 tablet_service.cc:1514] Processing DeleteTablet for tablet 6c324ba315d24d959c6acd106c03b042 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 09:01:07 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55114
I20260619 09:01:07.958341 2037 tablet_replica.cc:330] T 6c324ba315d24d959c6acd106c03b042 P 7a73cdd031584af38e35564c1e20b7c8: stopping tablet replica
I20260619 09:01:07.958362 1504 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50524:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260619 09:01:07.958431 2037 raft_consensus.cc:2229] T 6c324ba315d24d959c6acd106c03b042 P 7a73cdd031584af38e35564c1e20b7c8 [term 1 FOLLOWER]: Raft consensus shutting down.
W20260619 09:01:07.958494 1504 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 09:01:07.958516 2037 raft_consensus.cc:2258] T 6c324ba315d24d959c6acd106c03b042 P 7a73cdd031584af38e35564c1e20b7c8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 09:01:07.958644 2038 tablet_replica.cc:330] T 6c324ba315d24d959c6acd106c03b042 P 176dc359022b40eb964b3e93025673ac: stopping tablet replica
I20260619 09:01:07.958670 2039 tablet_replica.cc:330] T 6c324ba315d24d959c6acd106c03b042 P 374b600775294768a44620bdbaa906e7: stopping tablet replica
I20260619 09:01:07.958734 2038 raft_consensus.cc:2229] T 6c324ba315d24d959c6acd106c03b042 P 176dc359022b40eb964b3e93025673ac [term 1 LEADER]: Raft consensus shutting down.
I20260619 09:01:07.958763 2039 raft_consensus.cc:2229] T 6c324ba315d24d959c6acd106c03b042 P 374b600775294768a44620bdbaa906e7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 09:01:07.958807 2038 raft_consensus.cc:2258] T 6c324ba315d24d959c6acd106c03b042 P 176dc359022b40eb964b3e93025673ac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 09:01:07.958870 2039 raft_consensus.cc:2258] T 6c324ba315d24d959c6acd106c03b042 P 374b600775294768a44620bdbaa906e7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 09:01:07.959345 2038 ts_tablet_manager.cc:1905] T 6c324ba315d24d959c6acd106c03b042 P 176dc359022b40eb964b3e93025673ac: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 09:01:07.959630 1924 tablet_service.cc:1467] Processing CreateTablet for tablet edc8e2f62da145ebb4e684437f12ed66 (DEFAULT_TABLE table=test-table [id=79480f654c994d03ad767a67449f29f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 09:01:07.959669 2037 ts_tablet_manager.cc:1905] T 6c324ba315d24d959c6acd106c03b042 P 7a73cdd031584af38e35564c1e20b7c8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 09:01:07.959781 1642 tablet_service.cc:1467] Processing CreateTablet for tablet edc8e2f62da145ebb4e684437f12ed66 (DEFAULT_TABLE table=test-table [id=79480f654c994d03ad767a67449f29f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 09:01:07.959802 1783 tablet_service.cc:1467] Processing CreateTablet for tablet edc8e2f62da145ebb4e684437f12ed66 (DEFAULT_TABLE table=test-table [id=79480f654c994d03ad767a67449f29f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 09:01:07.959831 1924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet edc8e2f62da145ebb4e684437f12ed66. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 09:01:07.959867 1642 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet edc8e2f62da145ebb4e684437f12ed66. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 09:01:07.959887 1783 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet edc8e2f62da145ebb4e684437f12ed66. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 09:01:07.961087 2015 tablet_bootstrap.cc:492] T edc8e2f62da145ebb4e684437f12ed66 P 7a73cdd031584af38e35564c1e20b7c8: Bootstrap starting.
I20260619 09:01:07.961087 2014 tablet_bootstrap.cc:492] T edc8e2f62da145ebb4e684437f12ed66 P 374b600775294768a44620bdbaa906e7: Bootstrap starting.
I20260619 09:01:07.961110 2016 tablet_bootstrap.cc:492] T edc8e2f62da145ebb4e684437f12ed66 P 176dc359022b40eb964b3e93025673ac: Bootstrap starting.
I20260619 09:01:07.961457 2015 tablet_bootstrap.cc:654] T edc8e2f62da145ebb4e684437f12ed66 P 7a73cdd031584af38e35564c1e20b7c8: Neither blocks nor log segments found. Creating new log.
I20260619 09:01:07.961457 2014 tablet_bootstrap.cc:654] T edc8e2f62da145ebb4e684437f12ed66 P 374b600775294768a44620bdbaa906e7: Neither blocks nor log segments found. Creating new log.
I20260619 09:01:07.961503 2016 tablet_bootstrap.cc:654] T edc8e2f62da145ebb4e684437f12ed66 P 176dc359022b40eb964b3e93025673ac: Neither blocks nor log segments found. Creating new log.
I20260619 09:01:07.961915 2039 ts_tablet_manager.cc:1905] T 6c324ba315d24d959c6acd106c03b042 P 374b600775294768a44620bdbaa906e7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 09:01:07.962411 2038 ts_tablet_manager.cc:1918] T 6c324ba315d24d959c6acd106c03b042 P 176dc359022b40eb964b3e93025673ac: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 09:01:07.962479 2038 log.cc:1196] T 6c324ba315d24d959c6acd106c03b042 P 176dc359022b40eb964b3e93025673ac: Deleting WAL directory at /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781859658134510-392-0/minicluster-data/ts-2/wal/wals/6c324ba315d24d959c6acd106c03b042
I20260619 09:01:07.962683 2016 tablet_bootstrap.cc:492] T edc8e2f62da145ebb4e684437f12ed66 P 176dc359022b40eb964b3e93025673ac: No bootstrap required, opened a new log
I20260619 09:01:07.962729 2038 ts_tablet_manager.cc:1939] T 6c324ba315d24d959c6acd106c03b042 P 176dc359022b40eb964b3e93025673ac: Deleting consensus metadata
I20260619 09:01:07.962729 2016 ts_tablet_manager.cc:1397] T edc8e2f62da145ebb4e684437f12ed66 P 176dc359022b40eb964b3e93025673ac: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 09:01:07.963045 2016 raft_consensus.cc:348] T edc8e2f62da145ebb4e684437f12ed66 P 176dc359022b40eb964b3e93025673ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "374b600775294768a44620bdbaa906e7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32863 } } peers { permanent_uuid: "7a73cdd031584af38e35564c1e20b7c8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46601 } } peers { permanent_uuid: "176dc359022b40eb964b3e93025673ac" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33683 } }
I20260619 09:01:07.963117 2016 raft_consensus.cc:374] T edc8e2f62da145ebb4e684437f12ed66 P 176dc359022b40eb964b3e93025673ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 09:01:07.963136 2016 raft_consensus.cc:729] T edc8e2f62da145ebb4e684437f12ed66 P 176dc359022b40eb964b3e93025673ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 176dc359022b40eb964b3e93025673ac, State: Initialized, Role: FOLLOWER
I20260619 09:01:07.963178 2016 consensus_queue.cc:260] T edc8e2f62da145ebb4e684437f12ed66 P 176dc359022b40eb964b3e93025673ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "374b600775294768a44620bdbaa906e7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32863 } } peers { permanent_uuid: "7a73cdd031584af38e35564c1e20b7c8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46601 } } peers { permanent_uuid: "176dc359022b40eb964b3e93025673ac" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33683 } }
I20260619 09:01:07.963277 2016 ts_tablet_manager.cc:1428] T edc8e2f62da145ebb4e684437f12ed66 P 176dc359022b40eb964b3e93025673ac: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260619 09:01:07.963441 1488 catalog_manager.cc:4841] TS 176dc359022b40eb964b3e93025673ac (127.0.98.3:33683): tablet 6c324ba315d24d959c6acd106c03b042 (table test-table [id=0fec3e5848304167899560ff92c7b41b]) successfully deleted
I20260619 09:01:07.964000 2037 ts_tablet_manager.cc:1918] T 6c324ba315d24d959c6acd106c03b042 P 7a73cdd031584af38e35564c1e20b7c8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 09:01:07.964059 2037 log.cc:1196] T 6c324ba315d24d959c6acd106c03b042 P 7a73cdd031584af38e35564c1e20b7c8: Deleting WAL directory at /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781859658134510-392-0/minicluster-data/ts-1/wal/wals/6c324ba315d24d959c6acd106c03b042
I20260619 09:01:07.964483 2037 ts_tablet_manager.cc:1939] T 6c324ba315d24d959c6acd106c03b042 P 7a73cdd031584af38e35564c1e20b7c8: Deleting consensus metadata
I20260619 09:01:07.964749 2039 ts_tablet_manager.cc:1918] T 6c324ba315d24d959c6acd106c03b042 P 374b600775294768a44620bdbaa906e7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 09:01:07.964816 2039 log.cc:1196] T 6c324ba315d24d959c6acd106c03b042 P 374b600775294768a44620bdbaa906e7: Deleting WAL directory at /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781859658134510-392-0/minicluster-data/ts-0/wal/wals/6c324ba315d24d959c6acd106c03b042
I20260619 09:01:07.964825 1490 catalog_manager.cc:4841] TS 7a73cdd031584af38e35564c1e20b7c8 (127.0.98.2:46601): tablet 6c324ba315d24d959c6acd106c03b042 (table test-table [id=0fec3e5848304167899560ff92c7b41b]) successfully deleted
I20260619 09:01:07.965088 2039 ts_tablet_manager.cc:1939] T 6c324ba315d24d959c6acd106c03b042 P 374b600775294768a44620bdbaa906e7: Deleting consensus metadata
I20260619 09:01:07.965430 1490 catalog_manager.cc:4841] TS 374b600775294768a44620bdbaa906e7 (127.0.98.1:32863): tablet 6c324ba315d24d959c6acd106c03b042 (table test-table [id=0fec3e5848304167899560ff92c7b41b]) successfully deleted
I20260619 09:01:07.965762 2014 tablet_bootstrap.cc:492] T edc8e2f62da145ebb4e684437f12ed66 P 374b600775294768a44620bdbaa906e7: No bootstrap required, opened a new log
I20260619 09:01:07.965767 2015 tablet_bootstrap.cc:492] T edc8e2f62da145ebb4e684437f12ed66 P 7a73cdd031584af38e35564c1e20b7c8: No bootstrap required, opened a new log
I20260619 09:01:07.965811 2015 ts_tablet_manager.cc:1397] T edc8e2f62da145ebb4e684437f12ed66 P 7a73cdd031584af38e35564c1e20b7c8: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260619 09:01:07.965811 2014 ts_tablet_manager.cc:1397] T edc8e2f62da145ebb4e684437f12ed66 P 374b600775294768a44620bdbaa906e7: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260619 09:01:07.965965 2015 raft_consensus.cc:348] T edc8e2f62da145ebb4e684437f12ed66 P 7a73cdd031584af38e35564c1e20b7c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "374b600775294768a44620bdbaa906e7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32863 } } peers { permanent_uuid: "7a73cdd031584af38e35564c1e20b7c8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46601 } } peers { permanent_uuid: "176dc359022b40eb964b3e93025673ac" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33683 } }
I20260619 09:01:07.966032 2015 raft_consensus.cc:374] T edc8e2f62da145ebb4e684437f12ed66 P 7a73cdd031584af38e35564c1e20b7c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 09:01:07.966020 2014 raft_consensus.cc:348] T edc8e2f62da145ebb4e684437f12ed66 P 374b600775294768a44620bdbaa906e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "374b600775294768a44620bdbaa906e7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32863 } } peers { permanent_uuid: "7a73cdd031584af38e35564c1e20b7c8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46601 } } peers { permanent_uuid: "176dc359022b40eb964b3e93025673ac" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33683 } }
I20260619 09:01:07.966058 2015 raft_consensus.cc:729] T edc8e2f62da145ebb4e684437f12ed66 P 7a73cdd031584af38e35564c1e20b7c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a73cdd031584af38e35564c1e20b7c8, State: Initialized, Role: FOLLOWER
I20260619 09:01:07.966074 2014 raft_consensus.cc:374] T edc8e2f62da145ebb4e684437f12ed66 P 374b600775294768a44620bdbaa906e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 09:01:07.966097 2014 raft_consensus.cc:729] T edc8e2f62da145ebb4e684437f12ed66 P 374b600775294768a44620bdbaa906e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 374b600775294768a44620bdbaa906e7, State: Initialized, Role: FOLLOWER
I20260619 09:01:07.966097 2015 consensus_queue.cc:260] T edc8e2f62da145ebb4e684437f12ed66 P 7a73cdd031584af38e35564c1e20b7c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "374b600775294768a44620bdbaa906e7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32863 } } peers { permanent_uuid: "7a73cdd031584af38e35564c1e20b7c8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46601 } } peers { permanent_uuid: "176dc359022b40eb964b3e93025673ac" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33683 } }
I20260619 09:01:07.966130 2014 consensus_queue.cc:260] T edc8e2f62da145ebb4e684437f12ed66 P 374b600775294768a44620bdbaa906e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "374b600775294768a44620bdbaa906e7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32863 } } peers { permanent_uuid: "7a73cdd031584af38e35564c1e20b7c8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46601 } } peers { permanent_uuid: "176dc359022b40eb964b3e93025673ac" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33683 } }
I20260619 09:01:07.966200 2015 ts_tablet_manager.cc:1428] T edc8e2f62da145ebb4e684437f12ed66 P 7a73cdd031584af38e35564c1e20b7c8: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260619 09:01:07.966202 2014 ts_tablet_manager.cc:1428] T edc8e2f62da145ebb4e684437f12ed66 P 374b600775294768a44620bdbaa906e7: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
W20260619 09:01:08.013887 1709 tablet.cc:2369] T edc8e2f62da145ebb4e684437f12ed66 P 374b600775294768a44620bdbaa906e7: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 09:01:08.069917 1991 tablet.cc:2369] T edc8e2f62da145ebb4e684437f12ed66 P 176dc359022b40eb964b3e93025673ac: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 09:01:08.191012 2021 raft_consensus.cc:482] T edc8e2f62da145ebb4e684437f12ed66 P 374b600775294768a44620bdbaa906e7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 09:01:08.191099 2021 raft_consensus.cc:504] T edc8e2f62da145ebb4e684437f12ed66 P 374b600775294768a44620bdbaa906e7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "374b600775294768a44620bdbaa906e7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32863 } } peers { permanent_uuid: "7a73cdd031584af38e35564c1e20b7c8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46601 } } peers { permanent_uuid: "176dc359022b40eb964b3e93025673ac" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33683 } }
I20260619 09:01:08.191430 2021 leader_election.cc:288] T edc8e2f62da145ebb4e684437f12ed66 P 374b600775294768a44620bdbaa906e7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a73cdd031584af38e35564c1e20b7c8 (127.0.98.2:46601), 176dc359022b40eb964b3e93025673ac (127.0.98.3:33683)
I20260619 09:01:08.195360 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:08.191666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50465 (local address 127.0.98.2:46601)
0619 09:01:08.191724 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:08.191726 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:08.191831 (+ 105us) server_negotiation.cc:407] Connection header received
0619 09:01:08.191908 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:08.191909 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:08.191918 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:08.191967 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 09:01:08.192131 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:08.192859 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:08.194398 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:08.194991 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:08.195049 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:08.195060 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0619 09:01:08.195248 (+ 188us) server_negotiation.cc:299] Negotiation successful
0619 09:01:08.195299 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260619 09:01:08.195360 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:08.191702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43023 (local address 127.0.98.3:33683)
0619 09:01:08.191753 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:08.191755 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:08.191830 (+ 75us) server_negotiation.cc:407] Connection header received
0619 09:01:08.191896 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:08.191897 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:08.191907 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:08.191967 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 09:01:08.192117 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:08.192859 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:08.194397 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:08.194977 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:08.195049 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:08.195060 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0619 09:01:08.195248 (+ 188us) server_negotiation.cc:299] Negotiation successful
0619 09:01:08.195286 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260619 09:01:08.195422 2044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:08.191614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33683 (local address 127.0.98.1:43023)
0619 09:01:08.191767 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0619 09:01:08.191775 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0619 09:01:08.191834 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:01:08.191971 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:01:08.191972 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:01:08.191974 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 09:01:08.192062 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:08.192064 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:08.192944 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:08.192945 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:08.194303 (+ 1358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:08.194304 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:08.195084 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:08.195085 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:08.195168 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:08.195175 (+ 7us) client_negotiation.cc:769] Sending connection context
0619 09:01:08.195229 (+ 54us) client_negotiation.cc:240] Negotiation successful
0619 09:01:08.195280 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"mutex_wait_us":72,"thread_start_us":81,"threads_started":1}
I20260619 09:01:08.195429 2043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:08.191596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46601 (local address 127.0.98.1:50465)
0619 09:01:08.191753 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0619 09:01:08.191764 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 09:01:08.191835 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:01:08.191972 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:01:08.191972 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:01:08.191975 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 09:01:08.192071 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:08.192072 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:08.192961 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:08.192962 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:08.194295 (+ 1333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:08.194298 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:08.195084 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:08.195085 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:08.195168 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:08.195174 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 09:01:08.195229 (+ 55us) client_negotiation.cc:240] Negotiation successful
0619 09:01:08.195280 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"mutex_wait_us":78,"thread_start_us":74,"threads_started":1}
I20260619 09:01:08.195693 1940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edc8e2f62da145ebb4e684437f12ed66" candidate_uuid: "374b600775294768a44620bdbaa906e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "176dc359022b40eb964b3e93025673ac" is_pre_election: true
I20260619 09:01:08.195822 1940 raft_consensus.cc:2454] T edc8e2f62da145ebb4e684437f12ed66 P 176dc359022b40eb964b3e93025673ac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 374b600775294768a44620bdbaa906e7 in term 0.
I20260619 09:01:08.195945 1803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edc8e2f62da145ebb4e684437f12ed66" candidate_uuid: "374b600775294768a44620bdbaa906e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a73cdd031584af38e35564c1e20b7c8" is_pre_election: true
I20260619 09:01:08.196034 1803 raft_consensus.cc:2454] T edc8e2f62da145ebb4e684437f12ed66 P 7a73cdd031584af38e35564c1e20b7c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 374b600775294768a44620bdbaa906e7 in term 0.
I20260619 09:01:08.196136 1592 leader_election.cc:302] T edc8e2f62da145ebb4e684437f12ed66 P 374b600775294768a44620bdbaa906e7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 176dc359022b40eb964b3e93025673ac, 374b600775294768a44620bdbaa906e7; no voters:
I20260619 09:01:08.196276 2021 raft_consensus.cc:2781] T edc8e2f62da145ebb4e684437f12ed66 P 374b600775294768a44620bdbaa906e7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 09:01:08.196326 2021 raft_consensus.cc:482] T edc8e2f62da145ebb4e684437f12ed66 P 374b600775294768a44620bdbaa906e7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 09:01:08.196341 2021 raft_consensus.cc:3037] T edc8e2f62da145ebb4e684437f12ed66 P 374b600775294768a44620bdbaa906e7 [term 0 FOLLOWER]: Advancing to term 1
I20260619 09:01:08.196993 2021 raft_consensus.cc:504] T edc8e2f62da145ebb4e684437f12ed66 P 374b600775294768a44620bdbaa906e7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "374b600775294768a44620bdbaa906e7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32863 } } peers { permanent_uuid: "7a73cdd031584af38e35564c1e20b7c8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46601 } } peers { permanent_uuid: "176dc359022b40eb964b3e93025673ac" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33683 } }
I20260619 09:01:08.197105 2021 leader_election.cc:288] T edc8e2f62da145ebb4e684437f12ed66 P 374b600775294768a44620bdbaa906e7 [CANDIDATE]: Term 1 election: Requested vote from peers 7a73cdd031584af38e35564c1e20b7c8 (127.0.98.2:46601), 176dc359022b40eb964b3e93025673ac (127.0.98.3:33683)
I20260619 09:01:08.197263 1803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edc8e2f62da145ebb4e684437f12ed66" candidate_uuid: "374b600775294768a44620bdbaa906e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a73cdd031584af38e35564c1e20b7c8"
I20260619 09:01:08.197336 1803 raft_consensus.cc:3037] T edc8e2f62da145ebb4e684437f12ed66 P 7a73cdd031584af38e35564c1e20b7c8 [term 0 FOLLOWER]: Advancing to term 1
I20260619 09:01:08.197327 1940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edc8e2f62da145ebb4e684437f12ed66" candidate_uuid: "374b600775294768a44620bdbaa906e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "176dc359022b40eb964b3e93025673ac"
I20260619 09:01:08.197420 1940 raft_consensus.cc:3037] T edc8e2f62da145ebb4e684437f12ed66 P 176dc359022b40eb964b3e93025673ac [term 0 FOLLOWER]: Advancing to term 1
I20260619 09:01:08.197896 1803 raft_consensus.cc:2454] T edc8e2f62da145ebb4e684437f12ed66 P 7a73cdd031584af38e35564c1e20b7c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 374b600775294768a44620bdbaa906e7 in term 1.
I20260619 09:01:08.198055 1594 leader_election.cc:302] T edc8e2f62da145ebb4e684437f12ed66 P 374b600775294768a44620bdbaa906e7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 374b600775294768a44620bdbaa906e7, 7a73cdd031584af38e35564c1e20b7c8; no voters:
I20260619 09:01:08.198093 1940 raft_consensus.cc:2454] T edc8e2f62da145ebb4e684437f12ed66 P 176dc359022b40eb964b3e93025673ac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 374b600775294768a44620bdbaa906e7 in term 1.
I20260619 09:01:08.198158 2021 raft_consensus.cc:2781] T edc8e2f62da145ebb4e684437f12ed66 P 374b600775294768a44620bdbaa906e7 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 09:01:08.198304 2021 raft_consensus.cc:686] T edc8e2f62da145ebb4e684437f12ed66 P 374b600775294768a44620bdbaa906e7 [term 1 LEADER]: Becoming Leader. State: Replica: 374b600775294768a44620bdbaa906e7, State: Running, Role: LEADER
I20260619 09:01:08.198379 2021 consensus_queue.cc:237] T edc8e2f62da145ebb4e684437f12ed66 P 374b600775294768a44620bdbaa906e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "374b600775294768a44620bdbaa906e7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32863 } } peers { permanent_uuid: "7a73cdd031584af38e35564c1e20b7c8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46601 } } peers { permanent_uuid: "176dc359022b40eb964b3e93025673ac" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33683 } }
I20260619 09:01:08.199069 1503 catalog_manager.cc:5495] T edc8e2f62da145ebb4e684437f12ed66 P 374b600775294768a44620bdbaa906e7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 374b600775294768a44620bdbaa906e7 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "374b600775294768a44620bdbaa906e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "374b600775294768a44620bdbaa906e7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a73cdd031584af38e35564c1e20b7c8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "176dc359022b40eb964b3e93025673ac" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33683 } health_report { overall_health: UNKNOWN } } }
Jun 19 09:01:08 dist-test-slave-mtdr krb5kdc[1459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859665, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 09:01:08.282123 1883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:08.275642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50532 (local address 127.0.98.62:43583)
0619 09:01:08.275733 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:08.275735 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:08.275823 (+ 88us) server_negotiation.cc:407] Connection header received
0619 09:01:08.275924 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:08.275924 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:08.275932 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:08.275981 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:01:08.276447 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:08.276928 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:08.277586 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:08.277758 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:08.277811 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:08.279813 (+ 2002us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:01:08.279814 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:01:08.279815 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:01:08.279822 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:01:08.281091 (+ 1269us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:01:08.281423 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:01:08.281424 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:01:08.281425 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:01:08.281474 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:01:08.281620 (+ 146us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:01:08.281620 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:01:08.281621 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:01:08.281720 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:01:08.281774 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0619 09:01:08.281904 (+ 130us) server_negotiation.cc:299] Negotiation successful
0619 09:01:08.282039 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":67,"server-negotiator.queue_time_us":58}
I20260619 09:01:08.282123 2052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:08.275563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43583 (local address 127.0.0.1:50532)
0619 09:01:08.275744 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0619 09:01:08.275758 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 09:01:08.275831 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:01:08.275993 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:01:08.275994 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:01:08.276312 (+ 318us) client_negotiation.cc:189] Negotiated authn=SASL
0619 09:01:08.276357 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:08.276360 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:08.276996 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:08.276997 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:08.277513 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:08.277515 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:08.277814 (+ 299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:08.277815 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:08.277859 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:08.278304 (+ 445us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 09:01:08.278308 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 09:01:08.279732 (+ 1424us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 09:01:08.281174 (+ 1442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:01:08.281181 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:01:08.281183 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:01:08.281363 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:01:08.281524 (+ 161us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:01:08.281525 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:01:08.281525 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:01:08.281575 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:01:08.281770 (+ 195us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 09:01:08.281777 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 09:01:08.281822 (+ 45us) client_negotiation.cc:769] Sending connection context
0619 09:01:08.281878 (+ 56us) client_negotiation.cc:240] Negotiation successful
0619 09:01:08.281990 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":88,"threads_started":1}
I20260619 09:01:08.284060 1503 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50532:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260619 09:01:08.284260 1503 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 09:01:08.285547 1642 tablet_service.cc:1467] Processing CreateTablet for tablet cb09267abe1f40689cf010e5276db222 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=f68a7b1314f74c1ea35e8d4861948ded]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 09:01:08.285586 1924 tablet_service.cc:1467] Processing CreateTablet for tablet cb09267abe1f40689cf010e5276db222 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=f68a7b1314f74c1ea35e8d4861948ded]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 09:01:08.285707 1642 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb09267abe1f40689cf010e5276db222. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 09:01:08.285737 1924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb09267abe1f40689cf010e5276db222. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 09:01:08.285702 1783 tablet_service.cc:1467] Processing CreateTablet for tablet cb09267abe1f40689cf010e5276db222 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=f68a7b1314f74c1ea35e8d4861948ded]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 09:01:08.285820 1783 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb09267abe1f40689cf010e5276db222. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 09:01:08.286844 2016 tablet_bootstrap.cc:492] T cb09267abe1f40689cf010e5276db222 P 176dc359022b40eb964b3e93025673ac: Bootstrap starting.
I20260619 09:01:08.287147 2016 tablet_bootstrap.cc:654] T cb09267abe1f40689cf010e5276db222 P 176dc359022b40eb964b3e93025673ac: Neither blocks nor log segments found. Creating new log.
I20260619 09:01:08.287166 2015 tablet_bootstrap.cc:492] T cb09267abe1f40689cf010e5276db222 P 7a73cdd031584af38e35564c1e20b7c8: Bootstrap starting.
I20260619 09:01:08.287546 2015 tablet_bootstrap.cc:654] T cb09267abe1f40689cf010e5276db222 P 7a73cdd031584af38e35564c1e20b7c8: Neither blocks nor log segments found. Creating new log.
I20260619 09:01:08.287729 2016 tablet_bootstrap.cc:492] T cb09267abe1f40689cf010e5276db222 P 176dc359022b40eb964b3e93025673ac: No bootstrap required, opened a new log
I20260619 09:01:08.287768 2016 ts_tablet_manager.cc:1397] T cb09267abe1f40689cf010e5276db222 P 176dc359022b40eb964b3e93025673ac: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260619 09:01:08.287871 2014 tablet_bootstrap.cc:492] T cb09267abe1f40689cf010e5276db222 P 374b600775294768a44620bdbaa906e7: Bootstrap starting.
I20260619 09:01:08.287874 2016 raft_consensus.cc:348] T cb09267abe1f40689cf010e5276db222 P 176dc359022b40eb964b3e93025673ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "374b600775294768a44620bdbaa906e7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32863 } } peers { permanent_uuid: "7a73cdd031584af38e35564c1e20b7c8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46601 } } peers { permanent_uuid: "176dc359022b40eb964b3e93025673ac" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33683 } }
I20260619 09:01:08.287946 2016 raft_consensus.cc:374] T cb09267abe1f40689cf010e5276db222 P 176dc359022b40eb964b3e93025673ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 09:01:08.287964 2016 raft_consensus.cc:729] T cb09267abe1f40689cf010e5276db222 P 176dc359022b40eb964b3e93025673ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 176dc359022b40eb964b3e93025673ac, State: Initialized, Role: FOLLOWER
I20260619 09:01:08.288013 2016 consensus_queue.cc:260] T cb09267abe1f40689cf010e5276db222 P 176dc359022b40eb964b3e93025673ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "374b600775294768a44620bdbaa906e7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32863 } } peers { permanent_uuid: "7a73cdd031584af38e35564c1e20b7c8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46601 } } peers { permanent_uuid: "176dc359022b40eb964b3e93025673ac" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33683 } }
I20260619 09:01:08.288097 2016 ts_tablet_manager.cc:1428] T cb09267abe1f40689cf010e5276db222 P 176dc359022b40eb964b3e93025673ac: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260619 09:01:08.288151 2014 tablet_bootstrap.cc:654] T cb09267abe1f40689cf010e5276db222 P 374b600775294768a44620bdbaa906e7: Neither blocks nor log segments found. Creating new log.
I20260619 09:01:08.288123 2020 tablet_replica.cc:439] T cb09267abe1f40689cf010e5276db222 P 176dc359022b40eb964b3e93025673ac: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "374b600775294768a44620bdbaa906e7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32863 } } peers { permanent_uuid: "7a73cdd031584af38e35564c1e20b7c8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46601 } } peers { permanent_uuid: "176dc359022b40eb964b3e93025673ac" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33683 } } }
I20260619 09:01:08.288208 2020 tablet_replica.cc:442] T cb09267abe1f40689cf010e5276db222 P 176dc359022b40eb964b3e93025673ac: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 09:01:08.288735 2015 tablet_bootstrap.cc:492] T cb09267abe1f40689cf010e5276db222 P 7a73cdd031584af38e35564c1e20b7c8: No bootstrap required, opened a new log
I20260619 09:01:08.288746 2014 tablet_bootstrap.cc:492] T cb09267abe1f40689cf010e5276db222 P 374b600775294768a44620bdbaa906e7: No bootstrap required, opened a new log
I20260619 09:01:08.288782 2015 ts_tablet_manager.cc:1397] T cb09267abe1f40689cf010e5276db222 P 7a73cdd031584af38e35564c1e20b7c8: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 09:01:08.288787 2014 ts_tablet_manager.cc:1397] T cb09267abe1f40689cf010e5276db222 P 374b600775294768a44620bdbaa906e7: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260619 09:01:08.288908 2015 raft_consensus.cc:348] T cb09267abe1f40689cf010e5276db222 P 7a73cdd031584af38e35564c1e20b7c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "374b600775294768a44620bdbaa906e7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32863 } } peers { permanent_uuid: "7a73cdd031584af38e35564c1e20b7c8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46601 } } peers { permanent_uuid: "176dc359022b40eb964b3e93025673ac" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33683 } }
I20260619 09:01:08.288913 2014 raft_consensus.cc:348] T cb09267abe1f40689cf010e5276db222 P 374b600775294768a44620bdbaa906e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "374b600775294768a44620bdbaa906e7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32863 } } peers { permanent_uuid: "7a73cdd031584af38e35564c1e20b7c8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46601 } } peers { permanent_uuid: "176dc359022b40eb964b3e93025673ac" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33683 } }
I20260619 09:01:08.288961 2014 raft_consensus.cc:374] T cb09267abe1f40689cf010e5276db222 P 374b600775294768a44620bdbaa906e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 09:01:08.288962 2015 raft_consensus.cc:374] T cb09267abe1f40689cf010e5276db222 P 7a73cdd031584af38e35564c1e20b7c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 09:01:08.288973 2014 raft_consensus.cc:729] T cb09267abe1f40689cf010e5276db222 P 374b600775294768a44620bdbaa906e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 374b600775294768a44620bdbaa906e7, State: Initialized, Role: FOLLOWER
I20260619 09:01:08.288980 2015 raft_consensus.cc:729] T cb09267abe1f40689cf010e5276db222 P 7a73cdd031584af38e35564c1e20b7c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a73cdd031584af38e35564c1e20b7c8, State: Initialized, Role: FOLLOWER
I20260619 09:01:08.289003 2014 consensus_queue.cc:260] T cb09267abe1f40689cf010e5276db222 P 374b600775294768a44620bdbaa906e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "374b600775294768a44620bdbaa906e7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32863 } } peers { permanent_uuid: "7a73cdd031584af38e35564c1e20b7c8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46601 } } peers { permanent_uuid: "176dc359022b40eb964b3e93025673ac" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33683 } }
I20260619 09:01:08.289021 2015 consensus_queue.cc:260] T cb09267abe1f40689cf010e5276db222 P 7a73cdd031584af38e35564c1e20b7c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "374b600775294768a44620bdbaa906e7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32863 } } peers { permanent_uuid: "7a73cdd031584af38e35564c1e20b7c8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46601 } } peers { permanent_uuid: "176dc359022b40eb964b3e93025673ac" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33683 } }
I20260619 09:01:08.289085 2014 ts_tablet_manager.cc:1428] T cb09267abe1f40689cf010e5276db222 P 374b600775294768a44620bdbaa906e7: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260619 09:01:08.289103 2015 ts_tablet_manager.cc:1428] T cb09267abe1f40689cf010e5276db222 P 7a73cdd031584af38e35564c1e20b7c8: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260619 09:01:08.289208 2021 tablet_replica.cc:439] T cb09267abe1f40689cf010e5276db222 P 374b600775294768a44620bdbaa906e7: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "374b600775294768a44620bdbaa906e7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32863 } } peers { permanent_uuid: "7a73cdd031584af38e35564c1e20b7c8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46601 } } peers { permanent_uuid: "176dc359022b40eb964b3e93025673ac" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33683 } } }
I20260619 09:01:08.289232 2022 tablet_replica.cc:439] T cb09267abe1f40689cf010e5276db222 P 7a73cdd031584af38e35564c1e20b7c8: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "374b600775294768a44620bdbaa906e7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32863 } } peers { permanent_uuid: "7a73cdd031584af38e35564c1e20b7c8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46601 } } peers { permanent_uuid: "176dc359022b40eb964b3e93025673ac" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33683 } } }
I20260619 09:01:08.289273 2021 tablet_replica.cc:442] T cb09267abe1f40689cf010e5276db222 P 374b600775294768a44620bdbaa906e7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 09:01:08.289286 2022 tablet_replica.cc:442] T cb09267abe1f40689cf010e5276db222 P 7a73cdd031584af38e35564c1e20b7c8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 09:01:08.509898 2020 raft_consensus.cc:482] T cb09267abe1f40689cf010e5276db222 P 176dc359022b40eb964b3e93025673ac [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 09:01:08.510000 2020 raft_consensus.cc:504] T cb09267abe1f40689cf010e5276db222 P 176dc359022b40eb964b3e93025673ac [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "374b600775294768a44620bdbaa906e7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32863 } } peers { permanent_uuid: "7a73cdd031584af38e35564c1e20b7c8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46601 } } peers { permanent_uuid: "176dc359022b40eb964b3e93025673ac" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33683 } }
I20260619 09:01:08.510144 2020 leader_election.cc:288] T cb09267abe1f40689cf010e5276db222 P 176dc359022b40eb964b3e93025673ac [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 374b600775294768a44620bdbaa906e7 (127.0.98.1:32863), 7a73cdd031584af38e35564c1e20b7c8 (127.0.98.2:46601)
I20260619 09:01:08.510454 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb09267abe1f40689cf010e5276db222" candidate_uuid: "176dc359022b40eb964b3e93025673ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "374b600775294768a44620bdbaa906e7" is_pre_election: true
I20260619 09:01:08.510504 1803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb09267abe1f40689cf010e5276db222" candidate_uuid: "176dc359022b40eb964b3e93025673ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a73cdd031584af38e35564c1e20b7c8" is_pre_election: true
I20260619 09:01:08.510560 1662 raft_consensus.cc:2454] T cb09267abe1f40689cf010e5276db222 P 374b600775294768a44620bdbaa906e7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 176dc359022b40eb964b3e93025673ac in term 0.
I20260619 09:01:08.510596 1803 raft_consensus.cc:2454] T cb09267abe1f40689cf010e5276db222 P 7a73cdd031584af38e35564c1e20b7c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 176dc359022b40eb964b3e93025673ac in term 0.
I20260619 09:01:08.510776 1876 leader_election.cc:302] T cb09267abe1f40689cf010e5276db222 P 176dc359022b40eb964b3e93025673ac [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 176dc359022b40eb964b3e93025673ac, 7a73cdd031584af38e35564c1e20b7c8; no voters:
I20260619 09:01:08.510910 2020 raft_consensus.cc:2781] T cb09267abe1f40689cf010e5276db222 P 176dc359022b40eb964b3e93025673ac [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 09:01:08.510962 2020 raft_consensus.cc:482] T cb09267abe1f40689cf010e5276db222 P 176dc359022b40eb964b3e93025673ac [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 09:01:08.510982 2020 raft_consensus.cc:3037] T cb09267abe1f40689cf010e5276db222 P 176dc359022b40eb964b3e93025673ac [term 0 FOLLOWER]: Advancing to term 1
I20260619 09:01:08.511749 2020 raft_consensus.cc:504] T cb09267abe1f40689cf010e5276db222 P 176dc359022b40eb964b3e93025673ac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "374b600775294768a44620bdbaa906e7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32863 } } peers { permanent_uuid: "7a73cdd031584af38e35564c1e20b7c8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46601 } } peers { permanent_uuid: "176dc359022b40eb964b3e93025673ac" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33683 } }
I20260619 09:01:08.511857 2020 leader_election.cc:288] T cb09267abe1f40689cf010e5276db222 P 176dc359022b40eb964b3e93025673ac [CANDIDATE]: Term 1 election: Requested vote from peers 374b600775294768a44620bdbaa906e7 (127.0.98.1:32863), 7a73cdd031584af38e35564c1e20b7c8 (127.0.98.2:46601)
I20260619 09:01:08.512041 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb09267abe1f40689cf010e5276db222" candidate_uuid: "176dc359022b40eb964b3e93025673ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "374b600775294768a44620bdbaa906e7"
I20260619 09:01:08.512064 1803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb09267abe1f40689cf010e5276db222" candidate_uuid: "176dc359022b40eb964b3e93025673ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a73cdd031584af38e35564c1e20b7c8"
I20260619 09:01:08.512111 1662 raft_consensus.cc:3037] T cb09267abe1f40689cf010e5276db222 P 374b600775294768a44620bdbaa906e7 [term 0 FOLLOWER]: Advancing to term 1
I20260619 09:01:08.512126 1803 raft_consensus.cc:3037] T cb09267abe1f40689cf010e5276db222 P 7a73cdd031584af38e35564c1e20b7c8 [term 0 FOLLOWER]: Advancing to term 1
I20260619 09:01:08.512791 1803 raft_consensus.cc:2454] T cb09267abe1f40689cf010e5276db222 P 7a73cdd031584af38e35564c1e20b7c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 176dc359022b40eb964b3e93025673ac in term 1.
I20260619 09:01:08.512791 1662 raft_consensus.cc:2454] T cb09267abe1f40689cf010e5276db222 P 374b600775294768a44620bdbaa906e7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 176dc359022b40eb964b3e93025673ac in term 1.
I20260619 09:01:08.512980 1876 leader_election.cc:302] T cb09267abe1f40689cf010e5276db222 P 176dc359022b40eb964b3e93025673ac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 176dc359022b40eb964b3e93025673ac, 7a73cdd031584af38e35564c1e20b7c8; no voters:
I20260619 09:01:08.513125 2020 raft_consensus.cc:2781] T cb09267abe1f40689cf010e5276db222 P 176dc359022b40eb964b3e93025673ac [term 1 FOLLOWER]: Leader election won for term 1
I20260619 09:01:08.513224 2020 raft_consensus.cc:686] T cb09267abe1f40689cf010e5276db222 P 176dc359022b40eb964b3e93025673ac [term 1 LEADER]: Becoming Leader. State: Replica: 176dc359022b40eb964b3e93025673ac, State: Running, Role: LEADER
I20260619 09:01:08.513269 2020 consensus_queue.cc:237] T cb09267abe1f40689cf010e5276db222 P 176dc359022b40eb964b3e93025673ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "374b600775294768a44620bdbaa906e7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32863 } } peers { permanent_uuid: "7a73cdd031584af38e35564c1e20b7c8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46601 } } peers { permanent_uuid: "176dc359022b40eb964b3e93025673ac" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33683 } }
I20260619 09:01:08.513490 2056 tablet_replica.cc:439] T cb09267abe1f40689cf010e5276db222 P 176dc359022b40eb964b3e93025673ac: TxnStatusTablet state changed. Reason: New leader 176dc359022b40eb964b3e93025673ac. Latest consensus state: current_term: 1 leader_uuid: "176dc359022b40eb964b3e93025673ac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "374b600775294768a44620bdbaa906e7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32863 } } peers { permanent_uuid: "7a73cdd031584af38e35564c1e20b7c8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46601 } } peers { permanent_uuid: "176dc359022b40eb964b3e93025673ac" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33683 } } }
I20260619 09:01:08.513571 2056 tablet_replica.cc:442] T cb09267abe1f40689cf010e5276db222 P 176dc359022b40eb964b3e93025673ac: This TxnStatusTablet replica's current role is: LEADER
I20260619 09:01:08.513700 2058 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260619 09:01:08.513810 2058 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260619 09:01:08.513805 1504 catalog_manager.cc:5495] T cb09267abe1f40689cf010e5276db222 P 176dc359022b40eb964b3e93025673ac reported cstate change: term changed from 0 to 1, leader changed from <none> to 176dc359022b40eb964b3e93025673ac (127.0.98.3). New cstate: current_term: 1 leader_uuid: "176dc359022b40eb964b3e93025673ac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "374b600775294768a44620bdbaa906e7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a73cdd031584af38e35564c1e20b7c8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "176dc359022b40eb964b3e93025673ac" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33683 } health_report { overall_health: HEALTHY } } }
I20260619 09:01:08.572479 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:08.569496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46172 (local address 127.0.98.3:33683)
0619 09:01:08.569534 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:08.569536 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:08.569595 (+ 59us) server_negotiation.cc:407] Connection header received
0619 09:01:08.569680 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:08.569681 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:08.569692 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:08.569738 (+ 46us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 09:01:08.569854 (+ 116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:08.570348 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:08.571480 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:08.571786 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:08.571856 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:08.572136 (+ 280us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 09:01:08.572236 (+ 100us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 09:01:08.572297 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0619 09:01:08.572396 (+ 99us) server_negotiation.cc:299] Negotiation successful
0619 09:01:08.572417 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260619 09:01:08.572535 2052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:08.569463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33683 (local address 127.0.0.1:46172)
0619 09:01:08.569545 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0619 09:01:08.569556 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 09:01:08.569614 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:01:08.569750 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:01:08.569751 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:01:08.569752 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 09:01:08.569803 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:08.569805 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:08.570451 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:08.570452 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:08.571378 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:08.571381 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:08.571874 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:08.571875 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:08.571995 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:08.572020 (+ 25us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 09:01:08.572315 (+ 295us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 09:01:08.572325 (+ 10us) client_negotiation.cc:769] Sending connection context
0619 09:01:08.572387 (+ 62us) client_negotiation.cc:240] Negotiation successful
0619 09:01:08.572448 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260619 09:01:08.573755 1662 raft_consensus.cc:1261] T cb09267abe1f40689cf010e5276db222 P 374b600775294768a44620bdbaa906e7 [term 1 FOLLOWER]: Refusing update from remote peer 176dc359022b40eb964b3e93025673ac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 09:01:08.573765 1803 raft_consensus.cc:1261] T cb09267abe1f40689cf010e5276db222 P 7a73cdd031584af38e35564c1e20b7c8 [term 1 FOLLOWER]: Refusing update from remote peer 176dc359022b40eb964b3e93025673ac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 09:01:08.573994 2056 consensus_queue.cc:1035] T cb09267abe1f40689cf010e5276db222 P 176dc359022b40eb964b3e93025673ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "374b600775294768a44620bdbaa906e7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 09:01:08.574117 2020 consensus_queue.cc:1035] T cb09267abe1f40689cf010e5276db222 P 176dc359022b40eb964b3e93025673ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a73cdd031584af38e35564c1e20b7c8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 09:01:08.574880 2022 tablet_replica.cc:439] T cb09267abe1f40689cf010e5276db222 P 7a73cdd031584af38e35564c1e20b7c8: TxnStatusTablet state changed. Reason: New leader 176dc359022b40eb964b3e93025673ac. Latest consensus state: current_term: 1 leader_uuid: "176dc359022b40eb964b3e93025673ac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "374b600775294768a44620bdbaa906e7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32863 } } peers { permanent_uuid: "7a73cdd031584af38e35564c1e20b7c8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46601 } } peers { permanent_uuid: "176dc359022b40eb964b3e93025673ac" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33683 } } }
I20260619 09:01:08.574980 2022 tablet_replica.cc:442] T cb09267abe1f40689cf010e5276db222 P 7a73cdd031584af38e35564c1e20b7c8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 09:01:08.575034 2021 tablet_replica.cc:439] T cb09267abe1f40689cf010e5276db222 P 374b600775294768a44620bdbaa906e7: TxnStatusTablet state changed. Reason: New leader 176dc359022b40eb964b3e93025673ac. Latest consensus state: current_term: 1 leader_uuid: "176dc359022b40eb964b3e93025673ac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "374b600775294768a44620bdbaa906e7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32863 } } peers { permanent_uuid: "7a73cdd031584af38e35564c1e20b7c8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46601 } } peers { permanent_uuid: "176dc359022b40eb964b3e93025673ac" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33683 } } }
I20260619 09:01:08.575115 2021 tablet_replica.cc:442] T cb09267abe1f40689cf010e5276db222 P 374b600775294768a44620bdbaa906e7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 09:01:08.575932 2056 tablet_replica.cc:439] T cb09267abe1f40689cf010e5276db222 P 176dc359022b40eb964b3e93025673ac: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "176dc359022b40eb964b3e93025673ac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "374b600775294768a44620bdbaa906e7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32863 } } peers { permanent_uuid: "7a73cdd031584af38e35564c1e20b7c8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46601 } } peers { permanent_uuid: "176dc359022b40eb964b3e93025673ac" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33683 } } }
I20260619 09:01:08.576030 2020 tablet_replica.cc:439] T cb09267abe1f40689cf010e5276db222 P 176dc359022b40eb964b3e93025673ac: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "176dc359022b40eb964b3e93025673ac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "374b600775294768a44620bdbaa906e7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32863 } } peers { permanent_uuid: "7a73cdd031584af38e35564c1e20b7c8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46601 } } peers { permanent_uuid: "176dc359022b40eb964b3e93025673ac" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33683 } } }
I20260619 09:01:08.576058 2021 tablet_replica.cc:439] T cb09267abe1f40689cf010e5276db222 P 374b600775294768a44620bdbaa906e7: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "176dc359022b40eb964b3e93025673ac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "374b600775294768a44620bdbaa906e7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32863 } } peers { permanent_uuid: "7a73cdd031584af38e35564c1e20b7c8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46601 } } peers { permanent_uuid: "176dc359022b40eb964b3e93025673ac" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33683 } } }
I20260619 09:01:08.576116 2021 tablet_replica.cc:442] T cb09267abe1f40689cf010e5276db222 P 374b600775294768a44620bdbaa906e7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 09:01:08.576115 2020 tablet_replica.cc:442] T cb09267abe1f40689cf010e5276db222 P 176dc359022b40eb964b3e93025673ac: This TxnStatusTablet replica's current role is: LEADER
I20260619 09:01:08.576020 2056 tablet_replica.cc:442] T cb09267abe1f40689cf010e5276db222 P 176dc359022b40eb964b3e93025673ac: This TxnStatusTablet replica's current role is: LEADER
I20260619 09:01:08.576576 2022 tablet_replica.cc:439] T cb09267abe1f40689cf010e5276db222 P 7a73cdd031584af38e35564c1e20b7c8: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "176dc359022b40eb964b3e93025673ac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "374b600775294768a44620bdbaa906e7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32863 } } peers { permanent_uuid: "7a73cdd031584af38e35564c1e20b7c8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46601 } } peers { permanent_uuid: "176dc359022b40eb964b3e93025673ac" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33683 } } }
I20260619 09:01:08.576656 2022 tablet_replica.cc:442] T cb09267abe1f40689cf010e5276db222 P 7a73cdd031584af38e35564c1e20b7c8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 09:01:08.583504 2072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:08.580640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55130 (local address 127.0.98.1:32863)
0619 09:01:08.580744 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:08.580746 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:08.580755 (+ 9us) server_negotiation.cc:407] Connection header received
0619 09:01:08.580796 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:08.580797 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:08.580814 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:08.580878 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 09:01:08.581022 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:08.581490 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:08.582529 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:08.582922 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:08.582986 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:08.583184 (+ 198us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 09:01:08.583291 (+ 107us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 09:01:08.583348 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0619 09:01:08.583425 (+ 77us) server_negotiation.cc:299] Negotiation successful
0619 09:01:08.583443 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85,"thread_start_us":55,"threads_started":1}
I20260619 09:01:08.588939 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:08.586721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58817 (local address 127.0.98.3:33683)
0619 09:01:08.586763 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:08.586765 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:08.586909 (+ 144us) server_negotiation.cc:407] Connection header received
0619 09:01:08.586970 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:08.586970 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:08.586983 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:08.587033 (+ 50us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 09:01:08.587177 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:08.587616 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:08.588325 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:08.588504 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:08.588557 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:08.588716 (+ 159us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 09:01:08.588747 (+ 31us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 09:01:08.588786 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0619 09:01:08.588836 (+ 50us) server_negotiation.cc:299] Negotiation successful
0619 09:01:08.588858 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260619 09:01:08.588958 2075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:08.586662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33683 (local address 127.0.98.1:58817)
0619 09:01:08.586844 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0619 09:01:08.586855 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 09:01:08.586914 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:01:08.587038 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:01:08.587039 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:01:08.587040 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 09:01:08.587101 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:08.587103 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:08.587682 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:08.587684 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:08.588252 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:08.588254 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:08.588560 (+ 306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:08.588562 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:08.588622 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:08.588650 (+ 28us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 09:01:08.588794 (+ 144us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 09:01:08.588796 (+ 2us) client_negotiation.cc:769] Sending connection context
0619 09:01:08.588835 (+ 39us) client_negotiation.cc:240] Negotiation successful
0619 09:01:08.588879 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":91,"threads_started":1}
I20260619 09:01:08.591499 1803 raft_consensus.cc:1261] T edc8e2f62da145ebb4e684437f12ed66 P 7a73cdd031584af38e35564c1e20b7c8 [term 1 FOLLOWER]: Refusing update from remote peer 374b600775294768a44620bdbaa906e7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 09:01:08.591531 1940 raft_consensus.cc:1261] T edc8e2f62da145ebb4e684437f12ed66 P 176dc359022b40eb964b3e93025673ac [term 1 FOLLOWER]: Refusing update from remote peer 374b600775294768a44620bdbaa906e7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 09:01:08.591795 2045 consensus_queue.cc:1035] T edc8e2f62da145ebb4e684437f12ed66 P 374b600775294768a44620bdbaa906e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a73cdd031584af38e35564c1e20b7c8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 09:01:08.591948 2021 consensus_queue.cc:1035] T edc8e2f62da145ebb4e684437f12ed66 P 374b600775294768a44620bdbaa906e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "176dc359022b40eb964b3e93025673ac" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33683 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 09:01:08.601310 2072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:08.597975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53919 (local address 127.0.98.1:32863)
0619 09:01:08.598020 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:08.598022 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:08.599023 (+ 1001us) server_negotiation.cc:407] Connection header received
0619 09:01:08.599091 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:08.599092 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:08.599102 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:08.599152 (+ 50us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 09:01:08.599283 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:08.599856 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:08.600682 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:08.600885 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:08.600937 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:08.601082 (+ 145us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 09:01:08.601115 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 09:01:08.601157 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0619 09:01:08.601208 (+ 51us) server_negotiation.cc:299] Negotiation successful
0619 09:01:08.601229 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260619 09:01:08.601327 2081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:08.597816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:32863 (local address 127.0.98.3:53919)
0619 09:01:08.598946 (+ 1130us) negotiation.cc:107] Waiting for socket to connect
0619 09:01:08.598959 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 09:01:08.599023 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:01:08.599170 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:01:08.599171 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:01:08.599173 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 09:01:08.599225 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:08.599226 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:08.599923 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:08.599924 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:08.600605 (+ 681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:08.600613 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:08.600941 (+ 328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:08.600941 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:08.601002 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:08.601018 (+ 16us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 09:01:08.601166 (+ 148us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 09:01:08.601167 (+ 1us) client_negotiation.cc:769] Sending connection context
0619 09:01:08.601206 (+ 39us) client_negotiation.cc:240] Negotiation successful
0619 09:01:08.601248 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1074,"thread_start_us":73,"threads_started":1}
I20260619 09:01:08.615597 1504 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50524:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 09:01:08.615726 1504 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50524:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 09:01:08.616385 1504 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 30c28e2314774e6089287721c79bb73a: Sending DeleteTablet for 3 replicas of tablet edc8e2f62da145ebb4e684437f12ed66
I20260619 09:01:08.616645 1924 tablet_service.cc:1514] Processing DeleteTablet for tablet edc8e2f62da145ebb4e684437f12ed66 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 09:01:08 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46150
I20260619 09:01:08.616809 1783 tablet_service.cc:1514] Processing DeleteTablet for tablet edc8e2f62da145ebb4e684437f12ed66 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 09:01:08 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38596
I20260619 09:01:08.617129 1642 tablet_service.cc:1514] Processing DeleteTablet for tablet edc8e2f62da145ebb4e684437f12ed66 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 09:01:08 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55114
I20260619 09:01:08.617338 2087 tablet_replica.cc:330] T edc8e2f62da145ebb4e684437f12ed66 P 7a73cdd031584af38e35564c1e20b7c8: stopping tablet replica
I20260619 09:01:08.617398 2087 raft_consensus.cc:2229] T edc8e2f62da145ebb4e684437f12ed66 P 7a73cdd031584af38e35564c1e20b7c8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 09:01:08.617437 2087 raft_consensus.cc:2258] T edc8e2f62da145ebb4e684437f12ed66 P 7a73cdd031584af38e35564c1e20b7c8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 09:01:08.617717 2087 ts_tablet_manager.cc:1905] T edc8e2f62da145ebb4e684437f12ed66 P 7a73cdd031584af38e35564c1e20b7c8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 09:01:08.619469 2087 ts_tablet_manager.cc:1918] T edc8e2f62da145ebb4e684437f12ed66 P 7a73cdd031584af38e35564c1e20b7c8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 09:01:08.619529 2087 log.cc:1196] T edc8e2f62da145ebb4e684437f12ed66 P 7a73cdd031584af38e35564c1e20b7c8: Deleting WAL directory at /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781859658134510-392-0/minicluster-data/ts-1/wal/wals/edc8e2f62da145ebb4e684437f12ed66
I20260619 09:01:08.619746 2087 ts_tablet_manager.cc:1939] T edc8e2f62da145ebb4e684437f12ed66 P 7a73cdd031584af38e35564c1e20b7c8: Deleting consensus metadata
I20260619 09:01:08.620028 1490 catalog_manager.cc:4841] TS 7a73cdd031584af38e35564c1e20b7c8 (127.0.98.2:46601): tablet edc8e2f62da145ebb4e684437f12ed66 (table test-table [id=79480f654c994d03ad767a67449f29f7]) successfully deleted
I20260619 09:01:08.620172 2088 tablet_replica.cc:330] T edc8e2f62da145ebb4e684437f12ed66 P 374b600775294768a44620bdbaa906e7: stopping tablet replica
I20260619 09:01:08.620232 2088 raft_consensus.cc:2229] T edc8e2f62da145ebb4e684437f12ed66 P 374b600775294768a44620bdbaa906e7 [term 1 LEADER]: Raft consensus shutting down.
I20260619 09:01:08.620328 2088 raft_consensus.cc:2258] T edc8e2f62da145ebb4e684437f12ed66 P 374b600775294768a44620bdbaa906e7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 09:01:08.620560 2088 ts_tablet_manager.cc:1905] T edc8e2f62da145ebb4e684437f12ed66 P 374b600775294768a44620bdbaa906e7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 09:01:08.622114 2088 ts_tablet_manager.cc:1918] T edc8e2f62da145ebb4e684437f12ed66 P 374b600775294768a44620bdbaa906e7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 09:01:08.622174 2088 log.cc:1196] T edc8e2f62da145ebb4e684437f12ed66 P 374b600775294768a44620bdbaa906e7: Deleting WAL directory at /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781859658134510-392-0/minicluster-data/ts-0/wal/wals/edc8e2f62da145ebb4e684437f12ed66
I20260619 09:01:08.622381 2088 ts_tablet_manager.cc:1939] T edc8e2f62da145ebb4e684437f12ed66 P 374b600775294768a44620bdbaa906e7: Deleting consensus metadata
I20260619 09:01:08.622697 1490 catalog_manager.cc:4841] TS 374b600775294768a44620bdbaa906e7 (127.0.98.1:32863): tablet edc8e2f62da145ebb4e684437f12ed66 (table test-table [id=79480f654c994d03ad767a67449f29f7]) successfully deleted
I20260619 09:01:08.622864 2086 tablet_replica.cc:330] T edc8e2f62da145ebb4e684437f12ed66 P 176dc359022b40eb964b3e93025673ac: stopping tablet replica
I20260619 09:01:08.622926 2086 raft_consensus.cc:2229] T edc8e2f62da145ebb4e684437f12ed66 P 176dc359022b40eb964b3e93025673ac [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 09:01:08.622977 2086 raft_consensus.cc:2258] T edc8e2f62da145ebb4e684437f12ed66 P 176dc359022b40eb964b3e93025673ac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 09:01:08.623308 2086 ts_tablet_manager.cc:1905] T edc8e2f62da145ebb4e684437f12ed66 P 176dc359022b40eb964b3e93025673ac: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 09:01:08.624799 2086 ts_tablet_manager.cc:1918] T edc8e2f62da145ebb4e684437f12ed66 P 176dc359022b40eb964b3e93025673ac: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 09:01:08.624853 2086 log.cc:1196] T edc8e2f62da145ebb4e684437f12ed66 P 176dc359022b40eb964b3e93025673ac: Deleting WAL directory at /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781859658134510-392-0/minicluster-data/ts-2/wal/wals/edc8e2f62da145ebb4e684437f12ed66
Jun 19 09:01:08 dist-test-slave-mtdr krb5kdc[1459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859667, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260619 09:01:08.625036 2086 ts_tablet_manager.cc:1939] T edc8e2f62da145ebb4e684437f12ed66 P 176dc359022b40eb964b3e93025673ac: Deleting consensus metadata
I20260619 09:01:08.625546 1488 catalog_manager.cc:4841] TS 176dc359022b40eb964b3e93025673ac (127.0.98.3:33683): tablet edc8e2f62da145ebb4e684437f12ed66 (table test-table [id=79480f654c994d03ad767a67449f29f7]) successfully deleted
I20260619 09:01:08.629048 2072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:08.617843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55138 (local address 127.0.98.1:32863)
0619 09:01:08.617908 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:08.617910 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:08.618141 (+ 231us) server_negotiation.cc:407] Connection header received
0619 09:01:08.618232 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:08.618233 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:08.618244 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:08.618272 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:01:08.619021 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:08.619789 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:08.621214 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:08.621506 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:08.621548 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:08.625749 (+ 4201us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:01:08.625756 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:01:08.625757 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:01:08.625765 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:01:08.627717 (+ 1952us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:01:08.628107 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:01:08.628110 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:01:08.628111 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:01:08.628180 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:01:08.628363 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:01:08.628364 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:01:08.628364 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:01:08.628530 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:01:08.628650 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0619 09:01:08.628853 (+ 203us) server_negotiation.cc:299] Negotiation successful
0619 09:01:08.628935 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
W20260619 09:01:08.629647 1695 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55138
I20260619 09:01:08.638055 1883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:08.630739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50548 (local address 127.0.98.62:43583)
0619 09:01:08.630782 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:08.630784 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:08.630957 (+ 173us) server_negotiation.cc:407] Connection header received
0619 09:01:08.630984 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:08.630984 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:08.630997 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:08.631045 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:01:08.631663 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:08.632435 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:08.633402 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:08.633718 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:08.633786 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:08.635208 (+ 1422us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:01:08.635210 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:01:08.635210 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:01:08.635221 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:01:08.636946 (+ 1725us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:01:08.637300 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:01:08.637301 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:01:08.637301 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:01:08.637358 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:01:08.637541 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:01:08.637542 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:01:08.637542 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:01:08.637668 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:01:08.637758 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0619 09:01:08.637882 (+ 124us) server_negotiation.cc:299] Negotiation successful
0619 09:01:08.637954 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
W20260619 09:01:08.638616 1504 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50548
I20260619 09:01:08.639358 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ez5NB/build/release/bin/kudu with pid 1576
I20260619 09:01:08.644611 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ez5NB/build/release/bin/kudu with pid 1717
I20260619 09:01:08.648792 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ez5NB/build/release/bin/kudu with pid 1858
I20260619 09:01:08.653314 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ez5NB/build/release/bin/kudu with pid 1472
2026-06-19T09:01:08Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (6877 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781859658134510-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 09:01:08 dist-test-slave-mtdr krb5kdc[2095](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 09:01:08 dist-test-slave-mtdr krb5kdc[2095](info): set up 2 sockets
Jun 19 09:01:08 dist-test-slave-mtdr krb5kdc[2095](info): commencing operation
krb5kdc: starting...
W20260619 09:01:09.598320 392 mini_kdc.cc:121] Time spent starting KDC: real 0.926s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781859658134510-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781859658134510-392-0/krb5kdc/test-user.keytab.
Jun 19 09:01:09 dist-test-slave-mtdr krb5kdc[2095](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859669, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T09:01:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T09:01:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781859658134510-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781859658134510-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 09:01:09.709365 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ez5NB/build/release/bin/kudu
/tmp/dist-test-task4ez5NB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781859658134510-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781859658134510-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781859658134510-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781859658134510-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46507
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781859658134510-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44119
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46507
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781859658134510-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781859658134510-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781859658134510-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 09:01:09.787348 2108 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 09:01:09.787539 2108 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 09:01:09.787559 2108 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 09:01:09.788655 2108 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 09:01:09.788694 2108 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260619 09:01:09.788707 2108 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 09:01:09.788718 2108 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 09:01:09.788741 2108 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 09:01:09.788765 2108 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 09:01:09.790198 2108 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44119
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781859658134510-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781859658134510-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46507
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781859658134510-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46507
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781859658134510-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2108
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781859658134510-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 09:01:09.790426 2108 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 09:01:09.790637 2108 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 09:01:09.795047 2114 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:01:09.795110 2108 server_base.cc:1034] running on GCE node
W20260619 09:01:09.795058 2113 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:01:09.795073 2116 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:01:09.795491 2108 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 09:01:09.795742 2108 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 09:01:09.796880 2108 hybrid_clock.cc:648] HybridClock initialized: now 1781859669796864 us; error 31 us; skew 500 ppm
Jun 19 09:01:09 dist-test-slave-mtdr krb5kdc[2095](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859669, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 09:01:09.799253 2108 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 09:01:09.799633 2108 webserver.cc:460] Webserver started at http://127.0.98.62:33221/ using document root <none> and password file <none>
I20260619 09:01:09.799858 2108 fs_manager.cc:359] Metadata directory not provided
I20260619 09:01:09.799906 2108 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 09:01:09.800028 2108 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 09:01:09.801007 2108 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781859658134510-392-0/minicluster-data/master-0/data/instance:
uuid: "5b3e060ff3214f0a95cf22a183e6e061"
format_stamp: "Formatted at 2026-06-19 09:01:09 on dist-test-slave-mtdr"
server_key: "b46a8f1fa5798e7a422995153bff3631"
server_key_iv: "7417edbce40e0a428b43dc5abc72a5b4"
server_key_version: "encryptionkey@0"
I20260619 09:01:09.801321 2108 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781859658134510-392-0/minicluster-data/master-0/wal/instance:
uuid: "5b3e060ff3214f0a95cf22a183e6e061"
format_stamp: "Formatted at 2026-06-19 09:01:09 on dist-test-slave-mtdr"
server_key: "b46a8f1fa5798e7a422995153bff3631"
server_key_iv: "7417edbce40e0a428b43dc5abc72a5b4"
server_key_version: "encryptionkey@0"
I20260619 09:01:09.802558 2108 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260619 09:01:09.803256 2123 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 09:01:09.803396 2108 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260619 09:01:09.803479 2108 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781859658134510-392-0/minicluster-data/master-0/data,/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781859658134510-392-0/minicluster-data/master-0/wal
uuid: "5b3e060ff3214f0a95cf22a183e6e061"
format_stamp: "Formatted at 2026-06-19 09:01:09 on dist-test-slave-mtdr"
server_key: "b46a8f1fa5798e7a422995153bff3631"
server_key_iv: "7417edbce40e0a428b43dc5abc72a5b4"
server_key_version: "encryptionkey@0"
I20260619 09:01:09.803543 2108 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781859658134510-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781859658134510-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781859658134510-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 09:01:09.820542 2108 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 09:01:09.822571 2108 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 09:01:09.822705 2108 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 09:01:09.826977 2108 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46507
I20260619 09:01:09.827102 2185 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46507 every 8 connection(s)
I20260619 09:01:09.827350 2108 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781859658134510-392-0/minicluster-data/master-0/data/info.pb
I20260619 09:01:09.828007 2186 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 09:01:09.830205 2186 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b3e060ff3214f0a95cf22a183e6e061: Bootstrap starting.
I20260619 09:01:09.830752 2186 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5b3e060ff3214f0a95cf22a183e6e061: Neither blocks nor log segments found. Creating new log.
I20260619 09:01:09.831036 2186 log.cc:824] T 00000000000000000000000000000000 P 5b3e060ff3214f0a95cf22a183e6e061: Log is configured to *not* fsync() on all Append() calls
I20260619 09:01:09.831724 2186 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b3e060ff3214f0a95cf22a183e6e061: No bootstrap required, opened a new log
I20260619 09:01:09.833765 2186 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5b3e060ff3214f0a95cf22a183e6e061 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b3e060ff3214f0a95cf22a183e6e061" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46507 } }
I20260619 09:01:09.833894 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ez5NB/build/release/bin/kudu as pid 2108
I20260619 09:01:09.833956 2186 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5b3e060ff3214f0a95cf22a183e6e061 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 09:01:09.833983 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781859658134510-392-0/minicluster-data/master-0/wal/instance
I20260619 09:01:09.833989 2186 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5b3e060ff3214f0a95cf22a183e6e061 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b3e060ff3214f0a95cf22a183e6e061, State: Initialized, Role: FOLLOWER
I20260619 09:01:09.834142 392 external_mini_cluster.cc:1428] Setting key 9e40a5358f53a4506803bf3f11d51c1b
I20260619 09:01:09.834177 2186 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5b3e060ff3214f0a95cf22a183e6e061 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b3e060ff3214f0a95cf22a183e6e061" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46507 } }
I20260619 09:01:09.834275 2186 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5b3e060ff3214f0a95cf22a183e6e061 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 09:01:09.834304 2186 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5b3e060ff3214f0a95cf22a183e6e061 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 09:01:09.834345 2186 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5b3e060ff3214f0a95cf22a183e6e061 [term 0 FOLLOWER]: Advancing to term 1
I20260619 09:01:09.834980 2186 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5b3e060ff3214f0a95cf22a183e6e061 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b3e060ff3214f0a95cf22a183e6e061" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46507 } }
I20260619 09:01:09.835098 2189 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5b3e060ff3214f0a95cf22a183e6e061 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 09:01:09.835170 2189 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5b3e060ff3214f0a95cf22a183e6e061 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b3e060ff3214f0a95cf22a183e6e061" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46507 } }
I20260619 09:01:09.835470 2189 leader_election.cc:302] T 00000000000000000000000000000000 P 5b3e060ff3214f0a95cf22a183e6e061 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5b3e060ff3214f0a95cf22a183e6e061; no voters:
I20260619 09:01:09.835623 2189 leader_election.cc:288] T 00000000000000000000000000000000 P 5b3e060ff3214f0a95cf22a183e6e061 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers
I20260619 09:01:09.835086 2186 leader_election.cc:302] T 00000000000000000000000000000000 P 5b3e060ff3214f0a95cf22a183e6e061 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5b3e060ff3214f0a95cf22a183e6e061; no voters:
I20260619 09:01:09.835659 2192 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5b3e060ff3214f0a95cf22a183e6e061 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260619 09:01:09.835721 2186 leader_election.cc:288] T 00000000000000000000000000000000 P 5b3e060ff3214f0a95cf22a183e6e061 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 09:01:09.835722 2192 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5b3e060ff3214f0a95cf22a183e6e061 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 09:01:09.835799 2192 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5b3e060ff3214f0a95cf22a183e6e061 [term 1 FOLLOWER]: Advancing to term 2
I20260619 09:01:09.835956 2186 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5b3e060ff3214f0a95cf22a183e6e061 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 09:01:09.836501 2192 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5b3e060ff3214f0a95cf22a183e6e061 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b3e060ff3214f0a95cf22a183e6e061" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46507 } }
I20260619 09:01:09.836575 2192 leader_election.cc:302] T 00000000000000000000000000000000 P 5b3e060ff3214f0a95cf22a183e6e061 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5b3e060ff3214f0a95cf22a183e6e061; no voters:
I20260619 09:01:09.836673 2192 leader_election.cc:288] T 00000000000000000000000000000000 P 5b3e060ff3214f0a95cf22a183e6e061 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260619 09:01:09.836709 2192 raft_consensus.cc:2741] T 00000000000000000000000000000000 P 5b3e060ff3214f0a95cf22a183e6e061 [term 2 FOLLOWER]: Leader election decision vote started in defunct term 1: won
I20260619 09:01:09.836771 2192 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b3e060ff3214f0a95cf22a183e6e061 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b3e060ff3214f0a95cf22a183e6e061" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46507 } } }
I20260619 09:01:09.836825 2192 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b3e060ff3214f0a95cf22a183e6e061 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 09:01:09.836861 2192 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5b3e060ff3214f0a95cf22a183e6e061 [term 2 FOLLOWER]: Leader election won for term 2
I20260619 09:01:09.836890 2192 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5b3e060ff3214f0a95cf22a183e6e061 [term 2 LEADER]: Becoming Leader. State: Replica: 5b3e060ff3214f0a95cf22a183e6e061, State: Running, Role: LEADER
I20260619 09:01:09.836946 2192 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5b3e060ff3214f0a95cf22a183e6e061 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b3e060ff3214f0a95cf22a183e6e061" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46507 } }
I20260619 09:01:09.837275 2189 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b3e060ff3214f0a95cf22a183e6e061 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5b3e060ff3214f0a95cf22a183e6e061. Latest consensus state: current_term: 2 leader_uuid: "5b3e060ff3214f0a95cf22a183e6e061" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b3e060ff3214f0a95cf22a183e6e061" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46507 } } }
I20260619 09:01:09.837342 2189 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b3e060ff3214f0a95cf22a183e6e061 [sys.catalog]: This master's current role is: LEADER
W20260619 09:01:09.837812 2206 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5b3e060ff3214f0a95cf22a183e6e061: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 09:01:09.837867 2206 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 09:01:09.837911 2207 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 09:01:09.838058 2207 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 09:01:09.839576 2207 catalog_manager.cc:1347] Generated new cluster ID: ddf945e6cb3b4cdfaaad1d1dbce05701
I20260619 09:01:09.839624 2207 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 19 09:01:09 dist-test-slave-mtdr krb5kdc[2095](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859669, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 09:01:09.845554 2191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:09.834672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36382 (local address 127.0.98.62:46507)
0619 09:01:09.834888 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:09.834897 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:09.834924 (+ 27us) server_negotiation.cc:407] Connection header received
0619 09:01:09.835019 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:09.835025 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:09.835066 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:09.835205 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:01:09.835927 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:09.837291 (+ 1364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:09.838427 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:09.839039 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:09.839136 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:09.842069 (+ 2933us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:01:09.842073 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:01:09.842074 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:01:09.842085 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:01:09.843970 (+ 1885us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:01:09.844364 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:01:09.844367 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:01:09.844367 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:01:09.844476 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:01:09.844673 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:01:09.844674 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:01:09.844674 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:01:09.844963 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:01:09.845086 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0619 09:01:09.845223 (+ 137us) server_negotiation.cc:299] Negotiation successful
0619 09:01:09.845346 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":98,"threads_started":1}
I20260619 09:01:09.852984 2207 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 09:01:09.853477 2207 catalog_manager.cc:1504] Loading token signing keys...
I20260619 09:01:09.857350 2207 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5b3e060ff3214f0a95cf22a183e6e061: Generated new TSK 0
I20260619 09:01:09.857496 2207 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781859658134510-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781859658134510-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 09:01:09.888067 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ez5NB/build/release/bin/kudu
/tmp/dist-test-task4ez5NB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781859658134510-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781859658134510-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781859658134510-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781859658134510-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46507
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781859658134510-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44119
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781859658134510-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781859658134510-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781859658134510-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 09:01:09.979540 2213 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 09:01:09.979770 2213 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 09:01:09.979809 2213 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 09:01:09.981274 2213 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 09:01:09.981428 2213 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 09:01:09.981455 2213 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260619 09:01:09.981467 2213 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 09:01:09.981484 2213 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 09:01:09.983273 2213 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44119
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781859658134510-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781859658134510-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781859658134510-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781859658134510-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:46507
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2213
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781859658134510-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 09:01:09.983487 2213 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 09:01:09.983692 2213 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 09:01:09.988691 2219 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:01:09.988745 2218 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:01:09.989037 2213 server_base.cc:1034] running on GCE node
W20260619 09:01:09.988691 2221 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:01:09.989235 2213 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 09:01:09.989488 2213 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 09:01:09.990648 2213 hybrid_clock.cc:648] HybridClock initialized: now 1781859669990616 us; error 46 us; skew 500 ppm
Jun 19 09:01:09 dist-test-slave-mtdr krb5kdc[2095](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859669, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 09:01:09.993882 2213 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 09:01:09.994354 2213 webserver.cc:460] Webserver started at http://127.0.98.1:35341/ using document root <none> and password file <none>
I20260619 09:01:09.994570 2213 fs_manager.cc:359] Metadata directory not provided
I20260619 09:01:09.994621 2213 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 09:01:09.994745 2213 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 09:01:09.995796 2213 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781859658134510-392-0/minicluster-data/ts-0/data/instance:
uuid: "55a2824214a14a79afa7dccda9503fc5"
format_stamp: "Formatted at 2026-06-19 09:01:09 on dist-test-slave-mtdr"
server_key: "391ed60204ad759739c4484190ee8d17"
server_key_iv: "7550fa7e9a901d3bfa7ee57088dec33e"
server_key_version: "encryptionkey@0"
I20260619 09:01:09.996122 2213 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781859658134510-392-0/minicluster-data/ts-0/wal/instance:
uuid: "55a2824214a14a79afa7dccda9503fc5"
format_stamp: "Formatted at 2026-06-19 09:01:09 on dist-test-slave-mtdr"
server_key: "391ed60204ad759739c4484190ee8d17"
server_key_iv: "7550fa7e9a901d3bfa7ee57088dec33e"
server_key_version: "encryptionkey@0"
I20260619 09:01:09.997339 2213 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 09:01:09.998016 2228 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 09:01:09.998183 2213 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 09:01:09.998263 2213 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781859658134510-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781859658134510-392-0/minicluster-data/ts-0/wal
uuid: "55a2824214a14a79afa7dccda9503fc5"
format_stamp: "Formatted at 2026-06-19 09:01:09 on dist-test-slave-mtdr"
server_key: "391ed60204ad759739c4484190ee8d17"
server_key_iv: "7550fa7e9a901d3bfa7ee57088dec33e"
server_key_version: "encryptionkey@0"
I20260619 09:01:09.998323 2213 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781859658134510-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781859658134510-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781859658134510-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 09:01:10.016320 2213 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 09:01:10.018172 2213 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 09:01:10.018303 2213 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 09:01:10.018805 2213 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 09:01:10.018875 2213 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 09:01:10.018906 2213 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 09:01:10.018923 2213 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 09:01:10 dist-test-slave-mtdr krb5kdc[2095](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859669, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 09:01:10.025449 2213 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37703
I20260619 09:01:10.025519 2343 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37703 every 8 connection(s)
I20260619 09:01:10.025810 2213 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781859658134510-392-0/minicluster-data/ts-0/data/info.pb
I20260619 09:01:10.027734 2191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:10.019232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53741 (local address 127.0.98.62:46507)
0619 09:01:10.019280 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:10.019282 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:10.019504 (+ 222us) server_negotiation.cc:407] Connection header received
0619 09:01:10.020019 (+ 515us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:10.020020 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:10.020032 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:10.020079 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:01:10.020709 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:10.021201 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:10.022175 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:10.022394 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:10.022448 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:10.025253 (+ 2805us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:01:10.025254 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:01:10.025255 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:01:10.025264 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:01:10.026652 (+ 1388us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:01:10.027041 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:01:10.027042 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:01:10.027042 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:01:10.027092 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:01:10.027271 (+ 179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:01:10.027271 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:01:10.027272 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:01:10.027362 (+ 90us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:01:10.027403 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0619 09:01:10.027611 (+ 208us) server_negotiation.cc:299] Negotiation successful
0619 09:01:10.027672 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260619 09:01:10.027876 2237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:10.019244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46507 (local address 127.0.98.1:53741)
0619 09:01:10.019391 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0619 09:01:10.019425 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0619 09:01:10.019922 (+ 497us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:01:10.020144 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:01:10.020147 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:01:10.020435 (+ 288us) client_negotiation.cc:189] Negotiated authn=SASL
0619 09:01:10.020621 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:10.020626 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:10.021278 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:10.021279 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:10.022095 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:10.022096 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:10.022460 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:10.022461 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:10.022592 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:10.023082 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 09:01:10.023092 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 09:01:10.025164 (+ 2072us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 09:01:10.026718 (+ 1554us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:01:10.026722 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:01:10.026723 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:01:10.026988 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:01:10.027140 (+ 152us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:01:10.027141 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:01:10.027141 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:01:10.027233 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:01:10.027404 (+ 171us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 09:01:10.027407 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 09:01:10.027518 (+ 111us) client_negotiation.cc:769] Sending connection context
0619 09:01:10.027592 (+ 74us) client_negotiation.cc:240] Negotiation successful
0619 09:01:10.027701 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":87,"threads_started":1}
I20260619 09:01:10.028399 2344 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46507
I20260619 09:01:10.028471 2344 heartbeater.cc:464] Registering TS with master...
I20260619 09:01:10.028636 2344 heartbeater.cc:511] Master 127.0.98.62:46507 requested a full tablet report, sending...
I20260619 09:01:10.029100 2139 ts_manager.cc:194] Registered new tserver with Master: 55a2824214a14a79afa7dccda9503fc5 (127.0.98.1:37703)
I20260619 09:01:10.029826 2139 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:53741
I20260619 09:01:10.033519 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ez5NB/build/release/bin/kudu as pid 2213
I20260619 09:01:10.033622 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781859658134510-392-0/minicluster-data/ts-0/wal/instance
I20260619 09:01:10.033787 2191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:10.029313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59559 (local address 127.0.98.62:46507)
0619 09:01:10.029350 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:10.029352 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:10.029431 (+ 79us) server_negotiation.cc:407] Connection header received
0619 09:01:10.029480 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:10.029480 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:10.029497 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:10.029556 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:01:10.029795 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:10.030375 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:10.031107 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:10.031280 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:10.031328 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:10.031992 (+ 664us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:01:10.031993 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:01:10.031994 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:01:10.031998 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:01:10.032953 (+ 955us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:01:10.033207 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:01:10.033207 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:01:10.033208 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:01:10.033246 (+ 38us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:01:10.033371 (+ 125us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:01:10.033371 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:01:10.033371 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:01:10.033447 (+ 76us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:01:10.033496 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0619 09:01:10.033644 (+ 148us) server_negotiation.cc:299] Negotiation successful
0619 09:01:10.033716 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260619 09:01:10.033795 2350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:10.029241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46507 (local address 127.0.98.1:59559)
0619 09:01:10.029359 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0619 09:01:10.029368 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 09:01:10.029438 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:01:10.029566 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:01:10.029567 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:01:10.029695 (+ 128us) client_negotiation.cc:189] Negotiated authn=SASL
0619 09:01:10.029747 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:10.029748 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:10.030461 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:10.030462 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:10.031051 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:10.031053 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:10.031330 (+ 277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:10.031331 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:10.031389 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:10.031675 (+ 286us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 09:01:10.031678 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 09:01:10.031932 (+ 254us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 09:01:10.033019 (+ 1087us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:01:10.033020 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:01:10.033020 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:01:10.033160 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:01:10.033280 (+ 120us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:01:10.033280 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:01:10.033281 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:01:10.033318 (+ 37us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:01:10.033494 (+ 176us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 09:01:10.033495 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 09:01:10.033547 (+ 52us) client_negotiation.cc:769] Sending connection context
0619 09:01:10.033617 (+ 70us) client_negotiation.cc:240] Negotiation successful
0619 09:01:10.033701 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":86,"thread_start_us":66,"threads_started":1}
I20260619 09:01:10.033800 392 external_mini_cluster.cc:1428] Setting key 1334fc282e875fbd13ee626bbac4a73d
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781859658134510-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781859658134510-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 09:01:10.056876 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ez5NB/build/release/bin/kudu
/tmp/dist-test-task4ez5NB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781859658134510-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781859658134510-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781859658134510-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781859658134510-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46507
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781859658134510-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44119
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781859658134510-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781859658134510-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781859658134510-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 09:01:10.134755 2353 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 09:01:10.134982 2353 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 09:01:10.135008 2353 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 09:01:10.136132 2353 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 09:01:10.136168 2353 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 09:01:10.136186 2353 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260619 09:01:10.136196 2353 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 09:01:10.136212 2353 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 09:01:10.137650 2353 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44119
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781859658134510-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781859658134510-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781859658134510-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781859658134510-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:46507
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2353
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781859658134510-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 09:01:10.137837 2353 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 09:01:10.138000 2353 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 09:01:10.142483 2359 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:01:10.142542 2358 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:01:10.142717 2353 server_base.cc:1034] running on GCE node
W20260619 09:01:10.142542 2361 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:01:10.142970 2353 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 09:01:10.143190 2353 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 09:01:10.144340 2353 hybrid_clock.cc:648] HybridClock initialized: now 1781859670144328 us; error 29 us; skew 500 ppm
Jun 19 09:01:10 dist-test-slave-mtdr krb5kdc[2095](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859670, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 09:01:10.146600 2353 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 09:01:10.147027 2353 webserver.cc:460] Webserver started at http://127.0.98.2:40763/ using document root <none> and password file <none>
I20260619 09:01:10.147228 2353 fs_manager.cc:359] Metadata directory not provided
I20260619 09:01:10.147279 2353 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 09:01:10.147396 2353 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 09:01:10.148372 2353 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781859658134510-392-0/minicluster-data/ts-1/data/instance:
uuid: "1c7ed360640b4867afe317a02f50e811"
format_stamp: "Formatted at 2026-06-19 09:01:10 on dist-test-slave-mtdr"
server_key: "fdacacc8f4140e5c1ce7b56005ac2a1e"
server_key_iv: "63e9d9330e82f00166d2d427a58c594f"
server_key_version: "encryptionkey@0"
I20260619 09:01:10.148684 2353 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781859658134510-392-0/minicluster-data/ts-1/wal/instance:
uuid: "1c7ed360640b4867afe317a02f50e811"
format_stamp: "Formatted at 2026-06-19 09:01:10 on dist-test-slave-mtdr"
server_key: "fdacacc8f4140e5c1ce7b56005ac2a1e"
server_key_iv: "63e9d9330e82f00166d2d427a58c594f"
server_key_version: "encryptionkey@0"
I20260619 09:01:10.149839 2353 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 09:01:10.150516 2368 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 09:01:10.150660 2353 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 09:01:10.150743 2353 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781859658134510-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781859658134510-392-0/minicluster-data/ts-1/wal
uuid: "1c7ed360640b4867afe317a02f50e811"
format_stamp: "Formatted at 2026-06-19 09:01:10 on dist-test-slave-mtdr"
server_key: "fdacacc8f4140e5c1ce7b56005ac2a1e"
server_key_iv: "63e9d9330e82f00166d2d427a58c594f"
server_key_version: "encryptionkey@0"
I20260619 09:01:10.150804 2353 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781859658134510-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781859658134510-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781859658134510-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 09:01:10.176397 2353 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 09:01:10.178128 2353 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 09:01:10.178256 2353 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 09:01:10.178745 2353 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 09:01:10.178781 2353 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 09:01:10.178809 2353 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 09:01:10.178848 2353 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 09:01:10 dist-test-slave-mtdr krb5kdc[2095](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859670, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 09:01:10.185696 2353 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33217
I20260619 09:01:10.185746 2483 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33217 every 8 connection(s)
I20260619 09:01:10.186101 2353 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781859658134510-392-0/minicluster-data/ts-1/data/info.pb
I20260619 09:01:10.188401 2191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:10.179161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51189 (local address 127.0.98.62:46507)
0619 09:01:10.179225 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:10.179227 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:10.179426 (+ 199us) server_negotiation.cc:407] Connection header received
0619 09:01:10.180057 (+ 631us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:10.180058 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:10.180068 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:10.180126 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:01:10.180964 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:10.181706 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:10.183254 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:10.183548 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:10.183623 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:10.186068 (+ 2445us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:01:10.186069 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:01:10.186069 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:01:10.186075 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:01:10.187320 (+ 1245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:01:10.187656 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:01:10.187657 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:01:10.187657 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:01:10.187701 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:01:10.187872 (+ 171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:01:10.187872 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:01:10.187872 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:01:10.187960 (+ 88us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:01:10.188001 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0619 09:01:10.188272 (+ 271us) server_negotiation.cc:299] Negotiation successful
0619 09:01:10.188338 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260619 09:01:10.188498 2377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:10.179120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46507 (local address 127.0.98.2:51189)
0619 09:01:10.179289 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0619 09:01:10.179337 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0619 09:01:10.179951 (+ 614us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:01:10.180216 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:01:10.180222 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:01:10.180559 (+ 337us) client_negotiation.cc:189] Negotiated authn=SASL
0619 09:01:10.180838 (+ 279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:10.180846 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:10.181820 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:10.181821 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:10.183166 (+ 1345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:10.183170 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:10.183633 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:10.183634 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:10.183792 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:10.184246 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 09:01:10.184257 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 09:01:10.185978 (+ 1721us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 09:01:10.187382 (+ 1404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:01:10.187387 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:01:10.187389 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:01:10.187599 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:01:10.187745 (+ 146us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:01:10.187745 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:01:10.187745 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:01:10.187831 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:01:10.188021 (+ 190us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 09:01:10.188024 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 09:01:10.188132 (+ 108us) client_negotiation.cc:769] Sending connection context
0619 09:01:10.188212 (+ 80us) client_negotiation.cc:240] Negotiation successful
0619 09:01:10.188326 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":86,"threads_started":1}
I20260619 09:01:10.188925 2484 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46507
I20260619 09:01:10.189004 2484 heartbeater.cc:464] Registering TS with master...
I20260619 09:01:10.189165 2484 heartbeater.cc:511] Master 127.0.98.62:46507 requested a full tablet report, sending...
I20260619 09:01:10.189508 2139 ts_manager.cc:194] Registered new tserver with Master: 1c7ed360640b4867afe317a02f50e811 (127.0.98.2:33217)
I20260619 09:01:10.189867 2139 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:51189
I20260619 09:01:10.190995 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ez5NB/build/release/bin/kudu as pid 2353
I20260619 09:01:10.191068 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781859658134510-392-0/minicluster-data/ts-1/wal/instance
I20260619 09:01:10.191249 392 external_mini_cluster.cc:1428] Setting key d78686e2de3e247636cd9f4a2f860034
I20260619 09:01:10.195223 2191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:10.189903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:32891 (local address 127.0.98.62:46507)
0619 09:01:10.189943 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:10.189945 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:10.190035 (+ 90us) server_negotiation.cc:407] Connection header received
0619 09:01:10.190110 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:10.190111 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:10.190118 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:10.190160 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:01:10.190434 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:10.191002 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:10.191925 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:10.192123 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:10.192173 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:10.193010 (+ 837us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:01:10.193011 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:01:10.193011 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:01:10.193018 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:01:10.194199 (+ 1181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:01:10.194496 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:01:10.194497 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:01:10.194497 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:01:10.194547 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:01:10.194704 (+ 157us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:01:10.194705 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:01:10.194705 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:01:10.194875 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:01:10.194937 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0619 09:01:10.195067 (+ 130us) server_negotiation.cc:299] Negotiation successful
0619 09:01:10.195132 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260619 09:01:10.195240 2490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:10.189842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46507 (local address 127.0.98.2:32891)
0619 09:01:10.189984 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0619 09:01:10.189994 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 09:01:10.190058 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:01:10.190165 (+ 107us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:01:10.190166 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:01:10.190325 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0619 09:01:10.190373 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:10.190374 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:10.191157 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:10.191158 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:10.191853 (+ 695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:10.191855 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:10.192175 (+ 320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:10.192176 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:10.192248 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:10.192591 (+ 343us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 09:01:10.192595 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 09:01:10.192913 (+ 318us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 09:01:10.194274 (+ 1361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:01:10.194274 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:01:10.194275 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:01:10.194428 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:01:10.194599 (+ 171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:01:10.194599 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:01:10.194599 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:01:10.194654 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:01:10.194934 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 09:01:10.194935 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 09:01:10.194976 (+ 41us) client_negotiation.cc:769] Sending connection context
0619 09:01:10.195039 (+ 63us) client_negotiation.cc:240] Negotiation successful
0619 09:01:10.195134 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":67,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781859658134510-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781859658134510-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 09:01:10.213706 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ez5NB/build/release/bin/kudu
/tmp/dist-test-task4ez5NB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781859658134510-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781859658134510-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781859658134510-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781859658134510-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46507
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781859658134510-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44119
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781859658134510-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781859658134510-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781859658134510-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 09:01:10.290951 2493 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 09:01:10.291131 2493 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 09:01:10.291152 2493 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 09:01:10.292217 2493 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 09:01:10.292249 2493 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 09:01:10.292268 2493 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260619 09:01:10.292279 2493 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 09:01:10.292294 2493 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 09:01:10.293638 2493 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44119
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781859658134510-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781859658134510-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781859658134510-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781859658134510-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:46507
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2493
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781859658134510-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 09:01:10.293809 2493 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 09:01:10.293977 2493 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 09:01:10.299393 2501 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:01:10.299398 2493 server_base.cc:1034] running on GCE node
W20260619 09:01:10.299458 2498 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:01:10.299499 2499 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:01:10.299763 2493 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 09:01:10.300002 2493 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 09:01:10.301167 2493 hybrid_clock.cc:648] HybridClock initialized: now 1781859670301144 us; error 38 us; skew 500 ppm
Jun 19 09:01:10 dist-test-slave-mtdr krb5kdc[2095](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859670, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 09:01:10.308169 2493 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 09:01:10.308617 2493 webserver.cc:460] Webserver started at http://127.0.98.3:33039/ using document root <none> and password file <none>
I20260619 09:01:10.308810 2493 fs_manager.cc:359] Metadata directory not provided
I20260619 09:01:10.308849 2493 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 09:01:10.308938 2493 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 09:01:10.309922 2493 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781859658134510-392-0/minicluster-data/ts-2/data/instance:
uuid: "a018d10fc9e547ce9bfba8421bb8770f"
format_stamp: "Formatted at 2026-06-19 09:01:10 on dist-test-slave-mtdr"
server_key: "90c2f06c2b36c2a9e481ef4610326c2d"
server_key_iv: "039135d53c0ca346257a36725305771e"
server_key_version: "encryptionkey@0"
I20260619 09:01:10.310204 2493 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781859658134510-392-0/minicluster-data/ts-2/wal/instance:
uuid: "a018d10fc9e547ce9bfba8421bb8770f"
format_stamp: "Formatted at 2026-06-19 09:01:10 on dist-test-slave-mtdr"
server_key: "90c2f06c2b36c2a9e481ef4610326c2d"
server_key_iv: "039135d53c0ca346257a36725305771e"
server_key_version: "encryptionkey@0"
I20260619 09:01:10.311444 2493 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 09:01:10.312131 2508 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 09:01:10.312336 2493 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 09:01:10.312419 2493 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781859658134510-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781859658134510-392-0/minicluster-data/ts-2/wal
uuid: "a018d10fc9e547ce9bfba8421bb8770f"
format_stamp: "Formatted at 2026-06-19 09:01:10 on dist-test-slave-mtdr"
server_key: "90c2f06c2b36c2a9e481ef4610326c2d"
server_key_iv: "039135d53c0ca346257a36725305771e"
server_key_version: "encryptionkey@0"
I20260619 09:01:10.312475 2493 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781859658134510-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781859658134510-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781859658134510-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 09:01:10.340397 2493 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 09:01:10.342528 2493 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 09:01:10.342635 2493 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 09:01:10.343150 2493 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 09:01:10.343189 2493 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 09:01:10.343214 2493 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 09:01:10.343230 2493 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 09:01:10 dist-test-slave-mtdr krb5kdc[2095](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859670, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 09:01:10.350018 2493 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37317
I20260619 09:01:10.350111 2623 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37317 every 8 connection(s)
I20260619 09:01:10.350445 2493 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781859658134510-392-0/minicluster-data/ts-2/data/info.pb
I20260619 09:01:10.352880 2191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:10.343681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41723 (local address 127.0.98.62:46507)
0619 09:01:10.343743 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:10.343745 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:10.343927 (+ 182us) server_negotiation.cc:407] Connection header received
0619 09:01:10.344496 (+ 569us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:10.344498 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:10.344510 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:10.344597 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:01:10.345389 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:10.346126 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:10.347445 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:10.347718 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:10.347793 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:10.350422 (+ 2629us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:01:10.350423 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:01:10.350424 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:01:10.350430 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:01:10.351787 (+ 1357us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:01:10.352168 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:01:10.352169 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:01:10.352169 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:01:10.352223 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:01:10.352386 (+ 163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:01:10.352386 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:01:10.352386 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:01:10.352482 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:01:10.352527 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0619 09:01:10.352759 (+ 232us) server_negotiation.cc:299] Negotiation successful
0619 09:01:10.352817 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260619 09:01:10.353061 2517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:10.343652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46507 (local address 127.0.98.3:41723)
0619 09:01:10.343815 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0619 09:01:10.343856 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 09:01:10.344379 (+ 523us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:01:10.344679 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:01:10.344684 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:01:10.345007 (+ 323us) client_negotiation.cc:189] Negotiated authn=SASL
0619 09:01:10.345287 (+ 280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:10.345293 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:10.346229 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:10.346230 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:10.347364 (+ 1134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:10.347367 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:10.347860 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:10.347861 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:10.348046 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:10.348528 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 09:01:10.348539 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 09:01:10.350310 (+ 1771us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 09:01:10.351879 (+ 1569us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:01:10.351885 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:01:10.351887 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:01:10.352103 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:01:10.352261 (+ 158us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:01:10.352261 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:01:10.352262 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:01:10.352347 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:01:10.352526 (+ 179us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 09:01:10.352529 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 09:01:10.352659 (+ 130us) client_negotiation.cc:769] Sending connection context
0619 09:01:10.352741 (+ 82us) client_negotiation.cc:240] Negotiation successful
0619 09:01:10.352858 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":97,"threads_started":1}
I20260619 09:01:10.353542 2624 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46507
I20260619 09:01:10.353621 2624 heartbeater.cc:464] Registering TS with master...
I20260619 09:01:10.353785 2624 heartbeater.cc:511] Master 127.0.98.62:46507 requested a full tablet report, sending...
I20260619 09:01:10.354125 2140 ts_manager.cc:194] Registered new tserver with Master: a018d10fc9e547ce9bfba8421bb8770f (127.0.98.3:37317)
I20260619 09:01:10.354465 2140 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:41723
I20260619 09:01:10.358312 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ez5NB/build/release/bin/kudu as pid 2493
I20260619 09:01:10.358397 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781859658134510-392-0/minicluster-data/ts-2/wal/instance
I20260619 09:01:10.358578 392 external_mini_cluster.cc:1428] Setting key bae8da46011ce883ceabc56c3a184607
I20260619 09:01:10.359433 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 09:01:10.359814 2630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:10.354446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46507 (local address 127.0.98.3:48027)
0619 09:01:10.354582 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0619 09:01:10.354591 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 09:01:10.354647 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:01:10.354755 (+ 108us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:01:10.354756 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:01:10.354953 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0619 09:01:10.355004 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:10.355005 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:10.355682 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:10.355683 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:10.356237 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:10.356239 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:10.356569 (+ 330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:10.356570 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:10.356638 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:10.357082 (+ 444us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 09:01:10.357086 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 09:01:10.357385 (+ 299us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 09:01:10.358856 (+ 1471us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:01:10.358857 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:01:10.358858 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:01:10.359069 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:01:10.359264 (+ 195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:01:10.359265 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:01:10.359266 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:01:10.359344 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:01:10.359576 (+ 232us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 09:01:10.359576 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 09:01:10.359613 (+ 37us) client_negotiation.cc:769] Sending connection context
0619 09:01:10.359662 (+ 49us) client_negotiation.cc:240] Negotiation successful
0619 09:01:10.359741 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":67,"threads_started":1}
I20260619 09:01:10.359829 2191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:10.354519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48027 (local address 127.0.98.62:46507)
0619 09:01:10.354620 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:10.354622 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:10.354628 (+ 6us) server_negotiation.cc:407] Connection header received
0619 09:01:10.354699 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:10.354700 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:10.354708 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:10.354750 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:01:10.355068 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:10.355574 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:10.356309 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:10.356486 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:10.356538 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:10.357481 (+ 943us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:01:10.357483 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:01:10.357483 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:01:10.357494 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:01:10.358762 (+ 1268us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:01:10.359141 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:01:10.359142 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:01:10.359142 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:01:10.359211 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:01:10.359402 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:01:10.359403 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:01:10.359403 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:01:10.359525 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:01:10.359583 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0619 09:01:10.359683 (+ 100us) server_negotiation.cc:299] Negotiation successful
0619 09:01:10.359749 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
Jun 19 09:01:10 dist-test-slave-mtdr krb5kdc[2095](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859670, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 09:01:10 dist-test-slave-mtdr krb5kdc[2095](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859670, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 09:01:10.380820 2191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:10.374374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36394 (local address 127.0.98.62:46507)
0619 09:01:10.374429 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:10.374431 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:10.374439 (+ 8us) server_negotiation.cc:407] Connection header received
0619 09:01:10.374463 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:10.374464 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:10.374472 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:10.374524 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:01:10.375173 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:10.375677 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:10.376434 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:10.376619 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:10.376685 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:10.378651 (+ 1966us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:01:10.378653 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:01:10.378654 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:01:10.378660 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:01:10.379876 (+ 1216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:01:10.380139 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:01:10.380139 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:01:10.380140 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:01:10.380220 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:01:10.380412 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:01:10.380413 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:01:10.380413 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:01:10.380501 (+ 88us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:01:10.380569 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0619 09:01:10.380700 (+ 131us) server_negotiation.cc:299] Negotiation successful
0619 09:01:10.380754 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260619 09:01:10.382638 2140 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36394:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 09:01:10.384210 2140 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 09:01:10.390771 2643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:10.386452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37703 (local address 127.0.0.1:41594)
0619 09:01:10.386977 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0619 09:01:10.387015 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 09:01:10.387084 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:01:10.387268 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:01:10.387271 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:01:10.387274 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 09:01:10.387406 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:10.387409 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:10.388719 (+ 1310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:10.388720 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:10.389777 (+ 1057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:10.389779 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:10.390575 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:10.390575 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:10.390629 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:10.390632 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 09:01:10.390680 (+ 48us) client_negotiation.cc:240] Negotiation successful
0619 09:01:10.390710 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":490,"mutex_wait_us":2,"thread_start_us":103,"threads_started":1}
I20260619 09:01:10.390843 2644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:10.386630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41594 (local address 127.0.98.1:37703)
0619 09:01:10.386852 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:10.386857 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:10.387081 (+ 224us) server_negotiation.cc:407] Connection header received
0619 09:01:10.387146 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:10.387148 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:10.387160 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:10.387304 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 09:01:10.387518 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:10.388615 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:10.389857 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:10.390506 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:10.390578 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:10.390632 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0619 09:01:10.390709 (+ 77us) server_negotiation.cc:299] Negotiation successful
0619 09:01:10.390742 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":83,"threads_started":1}
I20260619 09:01:10.390985 2646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:10.386795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48254 (local address 127.0.98.2:33217)
0619 09:01:10.386945 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:10.386948 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:10.387242 (+ 294us) server_negotiation.cc:407] Connection header received
0619 09:01:10.387286 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:10.387288 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:10.387303 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:10.387369 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 09:01:10.387555 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:10.388260 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:10.389601 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:10.390380 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:10.390451 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:10.390513 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0619 09:01:10.390844 (+ 331us) server_negotiation.cc:299] Negotiation successful
0619 09:01:10.390882 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":88,"threads_started":1}
I20260619 09:01:10.391319 2642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:10.386457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37317 (local address 127.0.0.1:58520)
0619 09:01:10.387264 (+ 807us) negotiation.cc:107] Waiting for socket to connect
0619 09:01:10.387271 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0619 09:01:10.387311 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:01:10.387542 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:01:10.387542 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:01:10.387544 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 09:01:10.387579 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:10.387580 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:10.388766 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:10.388767 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:10.389700 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:10.389703 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:10.391092 (+ 1389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:10.391093 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:10.391149 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:10.391153 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 09:01:10.391203 (+ 50us) client_negotiation.cc:240] Negotiation successful
0619 09:01:10.391241 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":791,"mutex_wait_us":3,"thread_start_us":103,"threads_started":1}
I20260619 09:01:10.391366 2645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:10.386730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58520 (local address 127.0.98.3:37317)
0619 09:01:10.386960 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:10.386965 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:10.387362 (+ 397us) server_negotiation.cc:407] Connection header received
0619 09:01:10.387402 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:10.387404 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:10.387418 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:10.387482 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 09:01:10.387649 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:10.388688 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:10.389777 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:10.390506 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:10.390578 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:10.390632 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0619 09:01:10.391228 (+ 596us) server_negotiation.cc:299] Negotiation successful
0619 09:01:10.391270 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":196,"thread_start_us":102,"threads_started":1}
I20260619 09:01:10.390771 2641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:10.386453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33217 (local address 127.0.0.1:48254)
0619 09:01:10.387157 (+ 704us) negotiation.cc:107] Waiting for socket to connect
0619 09:01:10.387163 (+ 6us) client_negotiation.cc:174] Beginning negotiation
0619 09:01:10.387207 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:01:10.387443 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:01:10.387444 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:01:10.387446 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 09:01:10.387489 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:10.387490 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:10.388335 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:10.388336 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:10.389520 (+ 1184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:10.389523 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:10.390463 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:10.390464 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:10.390545 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:10.390559 (+ 14us) client_negotiation.cc:769] Sending connection context
0619 09:01:10.390626 (+ 67us) client_negotiation.cc:240] Negotiation successful
0619 09:01:10.390679 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":696,"mutex_wait_us":2,"thread_start_us":48,"threads_started":1}
I20260619 09:01:10.391710 2278 tablet_service.cc:1467] Processing CreateTablet for tablet 03c76fa0d5864de7bfd9bdebd6a1c11c (DEFAULT_TABLE table=test-table [id=72e1df6b899b4ec9ac0003e548c91d49]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 09:01:10.392076 2278 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 03c76fa0d5864de7bfd9bdebd6a1c11c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 09:01:10.392192 2418 tablet_service.cc:1467] Processing CreateTablet for tablet 03c76fa0d5864de7bfd9bdebd6a1c11c (DEFAULT_TABLE table=test-table [id=72e1df6b899b4ec9ac0003e548c91d49]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 09:01:10.392491 2418 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 03c76fa0d5864de7bfd9bdebd6a1c11c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 09:01:10.394726 2647 tablet_bootstrap.cc:492] T 03c76fa0d5864de7bfd9bdebd6a1c11c P 55a2824214a14a79afa7dccda9503fc5: Bootstrap starting.
I20260619 09:01:10.395380 2558 tablet_service.cc:1467] Processing CreateTablet for tablet 03c76fa0d5864de7bfd9bdebd6a1c11c (DEFAULT_TABLE table=test-table [id=72e1df6b899b4ec9ac0003e548c91d49]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 09:01:10.395515 2647 tablet_bootstrap.cc:654] T 03c76fa0d5864de7bfd9bdebd6a1c11c P 55a2824214a14a79afa7dccda9503fc5: Neither blocks nor log segments found. Creating new log.
I20260619 09:01:10.395678 2558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 03c76fa0d5864de7bfd9bdebd6a1c11c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 09:01:10.395781 2647 log.cc:824] T 03c76fa0d5864de7bfd9bdebd6a1c11c P 55a2824214a14a79afa7dccda9503fc5: Log is configured to *not* fsync() on all Append() calls
I20260619 09:01:10.396420 2647 tablet_bootstrap.cc:492] T 03c76fa0d5864de7bfd9bdebd6a1c11c P 55a2824214a14a79afa7dccda9503fc5: No bootstrap required, opened a new log
I20260619 09:01:10.396469 2647 ts_tablet_manager.cc:1397] T 03c76fa0d5864de7bfd9bdebd6a1c11c P 55a2824214a14a79afa7dccda9503fc5: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260619 09:01:10.397754 2647 raft_consensus.cc:348] T 03c76fa0d5864de7bfd9bdebd6a1c11c P 55a2824214a14a79afa7dccda9503fc5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c7ed360640b4867afe317a02f50e811" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33217 } } peers { permanent_uuid: "55a2824214a14a79afa7dccda9503fc5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37703 } } peers { permanent_uuid: "a018d10fc9e547ce9bfba8421bb8770f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37317 } }
I20260619 09:01:10.397881 2647 raft_consensus.cc:374] T 03c76fa0d5864de7bfd9bdebd6a1c11c P 55a2824214a14a79afa7dccda9503fc5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 09:01:10.397912 2647 raft_consensus.cc:729] T 03c76fa0d5864de7bfd9bdebd6a1c11c P 55a2824214a14a79afa7dccda9503fc5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55a2824214a14a79afa7dccda9503fc5, State: Initialized, Role: FOLLOWER
I20260619 09:01:10.397984 2650 tablet_bootstrap.cc:492] T 03c76fa0d5864de7bfd9bdebd6a1c11c P a018d10fc9e547ce9bfba8421bb8770f: Bootstrap starting.
I20260619 09:01:10.398032 2647 consensus_queue.cc:260] T 03c76fa0d5864de7bfd9bdebd6a1c11c P 55a2824214a14a79afa7dccda9503fc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c7ed360640b4867afe317a02f50e811" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33217 } } peers { permanent_uuid: "55a2824214a14a79afa7dccda9503fc5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37703 } } peers { permanent_uuid: "a018d10fc9e547ce9bfba8421bb8770f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37317 } }
I20260619 09:01:10.398213 2648 tablet_bootstrap.cc:492] T 03c76fa0d5864de7bfd9bdebd6a1c11c P 1c7ed360640b4867afe317a02f50e811: Bootstrap starting.
I20260619 09:01:10.398289 2647 ts_tablet_manager.cc:1428] T 03c76fa0d5864de7bfd9bdebd6a1c11c P 55a2824214a14a79afa7dccda9503fc5: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260619 09:01:10.398393 2344 heartbeater.cc:503] Master 127.0.98.62:46507 was elected leader, sending a full tablet report...
I20260619 09:01:10.398855 2648 tablet_bootstrap.cc:654] T 03c76fa0d5864de7bfd9bdebd6a1c11c P 1c7ed360640b4867afe317a02f50e811: Neither blocks nor log segments found. Creating new log.
I20260619 09:01:10.398869 2650 tablet_bootstrap.cc:654] T 03c76fa0d5864de7bfd9bdebd6a1c11c P a018d10fc9e547ce9bfba8421bb8770f: Neither blocks nor log segments found. Creating new log.
I20260619 09:01:10.399189 2650 log.cc:824] T 03c76fa0d5864de7bfd9bdebd6a1c11c P a018d10fc9e547ce9bfba8421bb8770f: Log is configured to *not* fsync() on all Append() calls
I20260619 09:01:10.399189 2648 log.cc:824] T 03c76fa0d5864de7bfd9bdebd6a1c11c P 1c7ed360640b4867afe317a02f50e811: Log is configured to *not* fsync() on all Append() calls
I20260619 09:01:10.399881 2650 tablet_bootstrap.cc:492] T 03c76fa0d5864de7bfd9bdebd6a1c11c P a018d10fc9e547ce9bfba8421bb8770f: No bootstrap required, opened a new log
I20260619 09:01:10.399951 2650 ts_tablet_manager.cc:1397] T 03c76fa0d5864de7bfd9bdebd6a1c11c P a018d10fc9e547ce9bfba8421bb8770f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 09:01:10.400063 2648 tablet_bootstrap.cc:492] T 03c76fa0d5864de7bfd9bdebd6a1c11c P 1c7ed360640b4867afe317a02f50e811: No bootstrap required, opened a new log
I20260619 09:01:10.400121 2648 ts_tablet_manager.cc:1397] T 03c76fa0d5864de7bfd9bdebd6a1c11c P 1c7ed360640b4867afe317a02f50e811: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 09:01:10.401888 2650 raft_consensus.cc:348] T 03c76fa0d5864de7bfd9bdebd6a1c11c P a018d10fc9e547ce9bfba8421bb8770f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c7ed360640b4867afe317a02f50e811" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33217 } } peers { permanent_uuid: "55a2824214a14a79afa7dccda9503fc5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37703 } } peers { permanent_uuid: "a018d10fc9e547ce9bfba8421bb8770f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37317 } }
I20260619 09:01:10.402060 2650 raft_consensus.cc:374] T 03c76fa0d5864de7bfd9bdebd6a1c11c P a018d10fc9e547ce9bfba8421bb8770f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 09:01:10.402092 2650 raft_consensus.cc:729] T 03c76fa0d5864de7bfd9bdebd6a1c11c P a018d10fc9e547ce9bfba8421bb8770f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a018d10fc9e547ce9bfba8421bb8770f, State: Initialized, Role: FOLLOWER
I20260619 09:01:10.402007 2648 raft_consensus.cc:348] T 03c76fa0d5864de7bfd9bdebd6a1c11c P 1c7ed360640b4867afe317a02f50e811 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c7ed360640b4867afe317a02f50e811" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33217 } } peers { permanent_uuid: "55a2824214a14a79afa7dccda9503fc5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37703 } } peers { permanent_uuid: "a018d10fc9e547ce9bfba8421bb8770f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37317 } }
I20260619 09:01:10.402128 2648 raft_consensus.cc:374] T 03c76fa0d5864de7bfd9bdebd6a1c11c P 1c7ed360640b4867afe317a02f50e811 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 09:01:10.402156 2648 raft_consensus.cc:729] T 03c76fa0d5864de7bfd9bdebd6a1c11c P 1c7ed360640b4867afe317a02f50e811 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c7ed360640b4867afe317a02f50e811, State: Initialized, Role: FOLLOWER
I20260619 09:01:10.402206 2650 consensus_queue.cc:260] T 03c76fa0d5864de7bfd9bdebd6a1c11c P a018d10fc9e547ce9bfba8421bb8770f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c7ed360640b4867afe317a02f50e811" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33217 } } peers { permanent_uuid: "55a2824214a14a79afa7dccda9503fc5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37703 } } peers { permanent_uuid: "a018d10fc9e547ce9bfba8421bb8770f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37317 } }
I20260619 09:01:10.402240 2648 consensus_queue.cc:260] T 03c76fa0d5864de7bfd9bdebd6a1c11c P 1c7ed360640b4867afe317a02f50e811 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c7ed360640b4867afe317a02f50e811" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33217 } } peers { permanent_uuid: "55a2824214a14a79afa7dccda9503fc5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37703 } } peers { permanent_uuid: "a018d10fc9e547ce9bfba8421bb8770f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37317 } }
I20260619 09:01:10.402482 2648 ts_tablet_manager.cc:1428] T 03c76fa0d5864de7bfd9bdebd6a1c11c P 1c7ed360640b4867afe317a02f50e811: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260619 09:01:10.402482 2650 ts_tablet_manager.cc:1428] T 03c76fa0d5864de7bfd9bdebd6a1c11c P a018d10fc9e547ce9bfba8421bb8770f: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260619 09:01:10.402532 2624 heartbeater.cc:503] Master 127.0.98.62:46507 was elected leader, sending a full tablet report...
I20260619 09:01:10.402608 2484 heartbeater.cc:503] Master 127.0.98.62:46507 was elected leader, sending a full tablet report...
W20260619 09:01:10.436774 2485 tablet.cc:2369] T 03c76fa0d5864de7bfd9bdebd6a1c11c P 1c7ed360640b4867afe317a02f50e811: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 09:01:10.526604 2345 tablet.cc:2369] T 03c76fa0d5864de7bfd9bdebd6a1c11c P 55a2824214a14a79afa7dccda9503fc5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 09:01:10.601260 2625 tablet.cc:2369] T 03c76fa0d5864de7bfd9bdebd6a1c11c P a018d10fc9e547ce9bfba8421bb8770f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 09:01:10.692152 2651 raft_consensus.cc:482] T 03c76fa0d5864de7bfd9bdebd6a1c11c P 55a2824214a14a79afa7dccda9503fc5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 09:01:10.692282 2651 raft_consensus.cc:504] T 03c76fa0d5864de7bfd9bdebd6a1c11c P 55a2824214a14a79afa7dccda9503fc5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c7ed360640b4867afe317a02f50e811" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33217 } } peers { permanent_uuid: "55a2824214a14a79afa7dccda9503fc5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37703 } } peers { permanent_uuid: "a018d10fc9e547ce9bfba8421bb8770f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37317 } }
I20260619 09:01:10.692651 2651 leader_election.cc:288] T 03c76fa0d5864de7bfd9bdebd6a1c11c P 55a2824214a14a79afa7dccda9503fc5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c7ed360640b4867afe317a02f50e811 (127.0.98.2:33217), a018d10fc9e547ce9bfba8421bb8770f (127.0.98.3:37317)
I20260619 09:01:10.695410 2657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:10.692831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33217 (local address 127.0.98.1:44293)
0619 09:01:10.692951 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0619 09:01:10.692961 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 09:01:10.693022 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:01:10.693140 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:01:10.693141 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:01:10.693143 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 09:01:10.693213 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:10.693214 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:10.693816 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:10.693817 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:10.694615 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:10.694617 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:10.695163 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:10.695164 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:10.695234 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:10.695238 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 09:01:10.695284 (+ 46us) client_negotiation.cc:240] Negotiation successful
0619 09:01:10.695325 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"mutex_wait_us":28,"thread_start_us":68,"threads_started":1}
I20260619 09:01:10.695399 2646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:10.692904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44293 (local address 127.0.98.2:33217)
0619 09:01:10.692954 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:10.692956 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:10.693005 (+ 49us) server_negotiation.cc:407] Connection header received
0619 09:01:10.693073 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:10.693074 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:10.693084 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:10.693132 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 09:01:10.693267 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:10.693748 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:10.694697 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:10.695097 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:10.695157 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:10.695167 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0619 09:01:10.695294 (+ 127us) server_negotiation.cc:299] Negotiation successful
0619 09:01:10.695337 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260619 09:01:10.695482 2645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:10.692953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37171 (local address 127.0.98.3:37317)
0619 09:01:10.692995 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:10.692997 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:10.693007 (+ 10us) server_negotiation.cc:407] Connection header received
0619 09:01:10.693076 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:10.693077 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:10.693087 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:10.693134 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 09:01:10.693272 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:10.693750 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:10.694773 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:10.695178 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:10.695243 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:10.695252 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0619 09:01:10.695381 (+ 129us) server_negotiation.cc:299] Negotiation successful
0619 09:01:10.695418 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260619 09:01:10.695710 2656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:10.692838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37317 (local address 127.0.98.1:37171)
0619 09:01:10.692960 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0619 09:01:10.692971 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 09:01:10.693024 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:01:10.693142 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:01:10.693143 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:01:10.693145 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 09:01:10.693219 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:10.693221 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:10.693832 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:10.693833 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:10.694676 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:10.694679 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:10.695249 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:10.695250 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:10.695323 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:10.695326 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 09:01:10.695371 (+ 45us) client_negotiation.cc:240] Negotiation successful
0619 09:01:10.695411 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"mutex_wait_us":57,"thread_start_us":52,"threads_started":1}
I20260619 09:01:10.695858 2438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03c76fa0d5864de7bfd9bdebd6a1c11c" candidate_uuid: "55a2824214a14a79afa7dccda9503fc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c7ed360640b4867afe317a02f50e811" is_pre_election: true
I20260619 09:01:10.696015 2438 raft_consensus.cc:2454] T 03c76fa0d5864de7bfd9bdebd6a1c11c P 1c7ed360640b4867afe317a02f50e811 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 55a2824214a14a79afa7dccda9503fc5 in term 0.
I20260619 09:01:10.695983 2578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03c76fa0d5864de7bfd9bdebd6a1c11c" candidate_uuid: "55a2824214a14a79afa7dccda9503fc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a018d10fc9e547ce9bfba8421bb8770f" is_pre_election: true
I20260619 09:01:10.696097 2578 raft_consensus.cc:2454] T 03c76fa0d5864de7bfd9bdebd6a1c11c P a018d10fc9e547ce9bfba8421bb8770f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 55a2824214a14a79afa7dccda9503fc5 in term 0.
I20260619 09:01:10.696229 2230 leader_election.cc:302] T 03c76fa0d5864de7bfd9bdebd6a1c11c P 55a2824214a14a79afa7dccda9503fc5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c7ed360640b4867afe317a02f50e811, 55a2824214a14a79afa7dccda9503fc5; no voters:
I20260619 09:01:10.696372 2651 raft_consensus.cc:2781] T 03c76fa0d5864de7bfd9bdebd6a1c11c P 55a2824214a14a79afa7dccda9503fc5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 09:01:10.696425 2651 raft_consensus.cc:482] T 03c76fa0d5864de7bfd9bdebd6a1c11c P 55a2824214a14a79afa7dccda9503fc5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 09:01:10.696439 2651 raft_consensus.cc:3037] T 03c76fa0d5864de7bfd9bdebd6a1c11c P 55a2824214a14a79afa7dccda9503fc5 [term 0 FOLLOWER]: Advancing to term 1
I20260619 09:01:10.697068 2651 raft_consensus.cc:504] T 03c76fa0d5864de7bfd9bdebd6a1c11c P 55a2824214a14a79afa7dccda9503fc5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c7ed360640b4867afe317a02f50e811" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33217 } } peers { permanent_uuid: "55a2824214a14a79afa7dccda9503fc5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37703 } } peers { permanent_uuid: "a018d10fc9e547ce9bfba8421bb8770f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37317 } }
I20260619 09:01:10.697201 2651 leader_election.cc:288] T 03c76fa0d5864de7bfd9bdebd6a1c11c P 55a2824214a14a79afa7dccda9503fc5 [CANDIDATE]: Term 1 election: Requested vote from peers 1c7ed360640b4867afe317a02f50e811 (127.0.98.2:33217), a018d10fc9e547ce9bfba8421bb8770f (127.0.98.3:37317)
I20260619 09:01:10.697364 2438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03c76fa0d5864de7bfd9bdebd6a1c11c" candidate_uuid: "55a2824214a14a79afa7dccda9503fc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c7ed360640b4867afe317a02f50e811"
I20260619 09:01:10.697350 2578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03c76fa0d5864de7bfd9bdebd6a1c11c" candidate_uuid: "55a2824214a14a79afa7dccda9503fc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a018d10fc9e547ce9bfba8421bb8770f"
I20260619 09:01:10.697438 2438 raft_consensus.cc:3037] T 03c76fa0d5864de7bfd9bdebd6a1c11c P 1c7ed360640b4867afe317a02f50e811 [term 0 FOLLOWER]: Advancing to term 1
I20260619 09:01:10.697438 2578 raft_consensus.cc:3037] T 03c76fa0d5864de7bfd9bdebd6a1c11c P a018d10fc9e547ce9bfba8421bb8770f [term 0 FOLLOWER]: Advancing to term 1
I20260619 09:01:10.698233 2578 raft_consensus.cc:2454] T 03c76fa0d5864de7bfd9bdebd6a1c11c P a018d10fc9e547ce9bfba8421bb8770f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 55a2824214a14a79afa7dccda9503fc5 in term 1.
I20260619 09:01:10.698233 2438 raft_consensus.cc:2454] T 03c76fa0d5864de7bfd9bdebd6a1c11c P 1c7ed360640b4867afe317a02f50e811 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 55a2824214a14a79afa7dccda9503fc5 in term 1.
I20260619 09:01:10.698393 2230 leader_election.cc:302] T 03c76fa0d5864de7bfd9bdebd6a1c11c P 55a2824214a14a79afa7dccda9503fc5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c7ed360640b4867afe317a02f50e811, 55a2824214a14a79afa7dccda9503fc5; no voters:
I20260619 09:01:10.698491 2651 raft_consensus.cc:2781] T 03c76fa0d5864de7bfd9bdebd6a1c11c P 55a2824214a14a79afa7dccda9503fc5 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 09:01:10.698587 2651 raft_consensus.cc:686] T 03c76fa0d5864de7bfd9bdebd6a1c11c P 55a2824214a14a79afa7dccda9503fc5 [term 1 LEADER]: Becoming Leader. State: Replica: 55a2824214a14a79afa7dccda9503fc5, State: Running, Role: LEADER
I20260619 09:01:10.698673 2651 consensus_queue.cc:237] T 03c76fa0d5864de7bfd9bdebd6a1c11c P 55a2824214a14a79afa7dccda9503fc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c7ed360640b4867afe317a02f50e811" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33217 } } peers { permanent_uuid: "55a2824214a14a79afa7dccda9503fc5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37703 } } peers { permanent_uuid: "a018d10fc9e547ce9bfba8421bb8770f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37317 } }
I20260619 09:01:10.699504 2139 catalog_manager.cc:5495] T 03c76fa0d5864de7bfd9bdebd6a1c11c P 55a2824214a14a79afa7dccda9503fc5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 55a2824214a14a79afa7dccda9503fc5 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "55a2824214a14a79afa7dccda9503fc5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c7ed360640b4867afe317a02f50e811" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33217 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "55a2824214a14a79afa7dccda9503fc5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37703 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a018d10fc9e547ce9bfba8421bb8770f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37317 } health_report { overall_health: UNKNOWN } } }
Jun 19 09:01:10 dist-test-slave-mtdr krb5kdc[2095](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859669, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 09:01:10.769937 2665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:10.763525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46507 (local address 127.0.0.1:36408)
0619 09:01:10.763660 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0619 09:01:10.763670 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 09:01:10.763738 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:01:10.763885 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:01:10.763885 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:01:10.764174 (+ 289us) client_negotiation.cc:189] Negotiated authn=SASL
0619 09:01:10.764221 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:10.764223 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:10.764893 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:10.764894 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:10.765420 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:10.765422 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:10.765801 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:10.765802 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:10.765858 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:10.766362 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 09:01:10.766368 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0619 09:01:10.767670 (+ 1302us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 09:01:10.768971 (+ 1301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:01:10.768994 (+ 23us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:01:10.768995 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:01:10.769189 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:01:10.769334 (+ 145us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:01:10.769335 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:01:10.769335 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:01:10.769385 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:01:10.769666 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 09:01:10.769670 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 09:01:10.769709 (+ 39us) client_negotiation.cc:769] Sending connection context
0619 09:01:10.769759 (+ 50us) client_negotiation.cc:240] Negotiation successful
0619 09:01:10.769844 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":63,"threads_started":1}
I20260619 09:01:10.769989 2191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:10.763588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36408 (local address 127.0.98.62:46507)
0619 09:01:10.763674 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:10.763676 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:10.763760 (+ 84us) server_negotiation.cc:407] Connection header received
0619 09:01:10.763788 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:10.763789 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:10.763799 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:10.763859 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:01:10.764300 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:10.764807 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:10.765533 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:10.765713 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:10.765776 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:10.767749 (+ 1973us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:01:10.767750 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:01:10.767750 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:01:10.767757 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:01:10.768890 (+ 1133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:01:10.769245 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:01:10.769246 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:01:10.769246 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:01:10.769289 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:01:10.769470 (+ 181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:01:10.769471 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:01:10.769471 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:01:10.769602 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:01:10.769678 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0619 09:01:10.769786 (+ 108us) server_negotiation.cc:299] Negotiation successful
0619 09:01:10.769889 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":44,"server-negotiator.queue_time_us":61}
I20260619 09:01:10.771804 2139 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36408:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260619 09:01:10.771997 2139 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 09:01:10.773324 2418 tablet_service.cc:1467] Processing CreateTablet for tablet 442eace1507040779557647f319df0ad (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=fb21e86eab8d48e1a37f1ee63e0a8f9b]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 09:01:10.773324 2278 tablet_service.cc:1467] Processing CreateTablet for tablet 442eace1507040779557647f319df0ad (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=fb21e86eab8d48e1a37f1ee63e0a8f9b]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 09:01:10.773367 2558 tablet_service.cc:1467] Processing CreateTablet for tablet 442eace1507040779557647f319df0ad (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=fb21e86eab8d48e1a37f1ee63e0a8f9b]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 09:01:10.773488 2558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 442eace1507040779557647f319df0ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 09:01:10.773490 2418 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 442eace1507040779557647f319df0ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 09:01:10.773490 2278 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 442eace1507040779557647f319df0ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 09:01:10.774592 2650 tablet_bootstrap.cc:492] T 442eace1507040779557647f319df0ad P a018d10fc9e547ce9bfba8421bb8770f: Bootstrap starting.
I20260619 09:01:10.774911 2648 tablet_bootstrap.cc:492] T 442eace1507040779557647f319df0ad P 1c7ed360640b4867afe317a02f50e811: Bootstrap starting.
I20260619 09:01:10.774910 2647 tablet_bootstrap.cc:492] T 442eace1507040779557647f319df0ad P 55a2824214a14a79afa7dccda9503fc5: Bootstrap starting.
I20260619 09:01:10.775130 2650 tablet_bootstrap.cc:654] T 442eace1507040779557647f319df0ad P a018d10fc9e547ce9bfba8421bb8770f: Neither blocks nor log segments found. Creating new log.
I20260619 09:01:10.775287 2648 tablet_bootstrap.cc:654] T 442eace1507040779557647f319df0ad P 1c7ed360640b4867afe317a02f50e811: Neither blocks nor log segments found. Creating new log.
I20260619 09:01:10.775317 2647 tablet_bootstrap.cc:654] T 442eace1507040779557647f319df0ad P 55a2824214a14a79afa7dccda9503fc5: Neither blocks nor log segments found. Creating new log.
I20260619 09:01:10.775807 2650 tablet_bootstrap.cc:492] T 442eace1507040779557647f319df0ad P a018d10fc9e547ce9bfba8421bb8770f: No bootstrap required, opened a new log
I20260619 09:01:10.775857 2650 ts_tablet_manager.cc:1397] T 442eace1507040779557647f319df0ad P a018d10fc9e547ce9bfba8421bb8770f: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260619 09:01:10.775997 2650 raft_consensus.cc:348] T 442eace1507040779557647f319df0ad P a018d10fc9e547ce9bfba8421bb8770f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55a2824214a14a79afa7dccda9503fc5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37703 } } peers { permanent_uuid: "1c7ed360640b4867afe317a02f50e811" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33217 } } peers { permanent_uuid: "a018d10fc9e547ce9bfba8421bb8770f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37317 } }
I20260619 09:01:10.776072 2650 raft_consensus.cc:374] T 442eace1507040779557647f319df0ad P a018d10fc9e547ce9bfba8421bb8770f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 09:01:10.776087 2648 tablet_bootstrap.cc:492] T 442eace1507040779557647f319df0ad P 1c7ed360640b4867afe317a02f50e811: No bootstrap required, opened a new log
I20260619 09:01:10.776100 2650 raft_consensus.cc:729] T 442eace1507040779557647f319df0ad P a018d10fc9e547ce9bfba8421bb8770f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a018d10fc9e547ce9bfba8421bb8770f, State: Initialized, Role: FOLLOWER
I20260619 09:01:10.776124 2648 ts_tablet_manager.cc:1397] T 442eace1507040779557647f319df0ad P 1c7ed360640b4867afe317a02f50e811: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260619 09:01:10.776144 2650 consensus_queue.cc:260] T 442eace1507040779557647f319df0ad P a018d10fc9e547ce9bfba8421bb8770f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55a2824214a14a79afa7dccda9503fc5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37703 } } peers { permanent_uuid: "1c7ed360640b4867afe317a02f50e811" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33217 } } peers { permanent_uuid: "a018d10fc9e547ce9bfba8421bb8770f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37317 } }
I20260619 09:01:10.776188 2647 tablet_bootstrap.cc:492] T 442eace1507040779557647f319df0ad P 55a2824214a14a79afa7dccda9503fc5: No bootstrap required, opened a new log
I20260619 09:01:10.776227 2647 ts_tablet_manager.cc:1397] T 442eace1507040779557647f319df0ad P 55a2824214a14a79afa7dccda9503fc5: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260619 09:01:10.776234 2650 ts_tablet_manager.cc:1428] T 442eace1507040779557647f319df0ad P a018d10fc9e547ce9bfba8421bb8770f: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260619 09:01:10.776252 2648 raft_consensus.cc:348] T 442eace1507040779557647f319df0ad P 1c7ed360640b4867afe317a02f50e811 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55a2824214a14a79afa7dccda9503fc5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37703 } } peers { permanent_uuid: "1c7ed360640b4867afe317a02f50e811" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33217 } } peers { permanent_uuid: "a018d10fc9e547ce9bfba8421bb8770f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37317 } }
I20260619 09:01:10.776304 2648 raft_consensus.cc:374] T 442eace1507040779557647f319df0ad P 1c7ed360640b4867afe317a02f50e811 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 09:01:10.776322 2648 raft_consensus.cc:729] T 442eace1507040779557647f319df0ad P 1c7ed360640b4867afe317a02f50e811 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c7ed360640b4867afe317a02f50e811, State: Initialized, Role: FOLLOWER
I20260619 09:01:10.776317 2655 tablet_replica.cc:439] T 442eace1507040779557647f319df0ad P a018d10fc9e547ce9bfba8421bb8770f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55a2824214a14a79afa7dccda9503fc5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37703 } } peers { permanent_uuid: "1c7ed360640b4867afe317a02f50e811" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33217 } } peers { permanent_uuid: "a018d10fc9e547ce9bfba8421bb8770f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37317 } } }
I20260619 09:01:10.776360 2648 consensus_queue.cc:260] T 442eace1507040779557647f319df0ad P 1c7ed360640b4867afe317a02f50e811 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55a2824214a14a79afa7dccda9503fc5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37703 } } peers { permanent_uuid: "1c7ed360640b4867afe317a02f50e811" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33217 } } peers { permanent_uuid: "a018d10fc9e547ce9bfba8421bb8770f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37317 } }
I20260619 09:01:10.776362 2647 raft_consensus.cc:348] T 442eace1507040779557647f319df0ad P 55a2824214a14a79afa7dccda9503fc5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55a2824214a14a79afa7dccda9503fc5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37703 } } peers { permanent_uuid: "1c7ed360640b4867afe317a02f50e811" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33217 } } peers { permanent_uuid: "a018d10fc9e547ce9bfba8421bb8770f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37317 } }
I20260619 09:01:10.776405 2655 tablet_replica.cc:442] T 442eace1507040779557647f319df0ad P a018d10fc9e547ce9bfba8421bb8770f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 09:01:10.776432 2647 raft_consensus.cc:374] T 442eace1507040779557647f319df0ad P 55a2824214a14a79afa7dccda9503fc5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 09:01:10.776451 2647 raft_consensus.cc:729] T 442eace1507040779557647f319df0ad P 55a2824214a14a79afa7dccda9503fc5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55a2824214a14a79afa7dccda9503fc5, State: Initialized, Role: FOLLOWER
I20260619 09:01:10.776459 2648 ts_tablet_manager.cc:1428] T 442eace1507040779557647f319df0ad P 1c7ed360640b4867afe317a02f50e811: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260619 09:01:10.776489 2647 consensus_queue.cc:260] T 442eace1507040779557647f319df0ad P 55a2824214a14a79afa7dccda9503fc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55a2824214a14a79afa7dccda9503fc5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37703 } } peers { permanent_uuid: "1c7ed360640b4867afe317a02f50e811" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33217 } } peers { permanent_uuid: "a018d10fc9e547ce9bfba8421bb8770f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37317 } }
I20260619 09:01:10.776571 2647 ts_tablet_manager.cc:1428] T 442eace1507040779557647f319df0ad P 55a2824214a14a79afa7dccda9503fc5: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260619 09:01:10.776583 2654 tablet_replica.cc:439] T 442eace1507040779557647f319df0ad P 1c7ed360640b4867afe317a02f50e811: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55a2824214a14a79afa7dccda9503fc5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37703 } } peers { permanent_uuid: "1c7ed360640b4867afe317a02f50e811" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33217 } } peers { permanent_uuid: "a018d10fc9e547ce9bfba8421bb8770f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37317 } } }
I20260619 09:01:10.776645 2654 tablet_replica.cc:442] T 442eace1507040779557647f319df0ad P 1c7ed360640b4867afe317a02f50e811: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 09:01:10.776829 2651 tablet_replica.cc:439] T 442eace1507040779557647f319df0ad P 55a2824214a14a79afa7dccda9503fc5: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55a2824214a14a79afa7dccda9503fc5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37703 } } peers { permanent_uuid: "1c7ed360640b4867afe317a02f50e811" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33217 } } peers { permanent_uuid: "a018d10fc9e547ce9bfba8421bb8770f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37317 } } }
I20260619 09:01:10.776913 2651 tablet_replica.cc:442] T 442eace1507040779557647f319df0ad P 55a2824214a14a79afa7dccda9503fc5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 09:01:10.811349 2654 raft_consensus.cc:482] T 442eace1507040779557647f319df0ad P 1c7ed360640b4867afe317a02f50e811 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 09:01:10.811441 2654 raft_consensus.cc:504] T 442eace1507040779557647f319df0ad P 1c7ed360640b4867afe317a02f50e811 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55a2824214a14a79afa7dccda9503fc5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37703 } } peers { permanent_uuid: "1c7ed360640b4867afe317a02f50e811" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33217 } } peers { permanent_uuid: "a018d10fc9e547ce9bfba8421bb8770f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37317 } }
I20260619 09:01:10.811765 2654 leader_election.cc:288] T 442eace1507040779557647f319df0ad P 1c7ed360640b4867afe317a02f50e811 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 55a2824214a14a79afa7dccda9503fc5 (127.0.98.1:37703), a018d10fc9e547ce9bfba8421bb8770f (127.0.98.3:37317)
I20260619 09:01:10.814450 2644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:10.811993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51357 (local address 127.0.98.1:37703)
0619 09:01:10.812039 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:10.812041 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:10.812113 (+ 72us) server_negotiation.cc:407] Connection header received
0619 09:01:10.812204 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:10.812205 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:10.812216 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:10.812263 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 09:01:10.812397 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:10.812853 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:10.813798 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:10.814153 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:10.814212 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:10.814223 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0619 09:01:10.814341 (+ 118us) server_negotiation.cc:299] Negotiation successful
0619 09:01:10.814379 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260619 09:01:10.814476 2670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:10.811935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37703 (local address 127.0.98.2:51357)
0619 09:01:10.812058 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0619 09:01:10.812067 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 09:01:10.812138 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:01:10.812270 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:01:10.812271 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:01:10.812272 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 09:01:10.812342 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:10.812343 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:10.812922 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:10.812923 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:10.813719 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:10.813721 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:10.814221 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:10.814222 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:10.814284 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:10.814288 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 09:01:10.814332 (+ 44us) client_negotiation.cc:240] Negotiation successful
0619 09:01:10.814369 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":58,"threads_started":1}
I20260619 09:01:10.814713 2298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "442eace1507040779557647f319df0ad" candidate_uuid: "1c7ed360640b4867afe317a02f50e811" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "55a2824214a14a79afa7dccda9503fc5" is_pre_election: true
I20260619 09:01:10.814854 2298 raft_consensus.cc:2454] T 442eace1507040779557647f319df0ad P 55a2824214a14a79afa7dccda9503fc5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c7ed360640b4867afe317a02f50e811 in term 0.
I20260619 09:01:10.814857 2669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:10.811919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37317 (local address 127.0.98.2:51321)
0619 09:01:10.812054 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0619 09:01:10.812063 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 09:01:10.812115 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:01:10.812242 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:01:10.812242 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:01:10.812244 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 09:01:10.812312 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:10.812313 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:10.812925 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:10.812926 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:10.814070 (+ 1144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:10.814072 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:10.814595 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:10.814596 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:10.814664 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:10.814667 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 09:01:10.814710 (+ 43us) client_negotiation.cc:240] Negotiation successful
0619 09:01:10.814745 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"mutex_wait_us":39,"thread_start_us":74,"threads_started":1}
I20260619 09:01:10.814854 2645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:10.812050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51321 (local address 127.0.98.3:37317)
0619 09:01:10.812095 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:10.812097 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:10.812105 (+ 8us) server_negotiation.cc:407] Connection header received
0619 09:01:10.812180 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:10.812181 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:10.812189 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:10.812233 (+ 44us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 09:01:10.812392 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:10.812860 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:10.814155 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:10.814493 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:10.814552 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:10.814561 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0619 09:01:10.814741 (+ 180us) server_negotiation.cc:299] Negotiation successful
0619 09:01:10.814774 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260619 09:01:10.815048 2369 leader_election.cc:302] T 442eace1507040779557647f319df0ad P 1c7ed360640b4867afe317a02f50e811 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c7ed360640b4867afe317a02f50e811, 55a2824214a14a79afa7dccda9503fc5; no voters:
I20260619 09:01:10.815076 2578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "442eace1507040779557647f319df0ad" candidate_uuid: "1c7ed360640b4867afe317a02f50e811" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a018d10fc9e547ce9bfba8421bb8770f" is_pre_election: true
I20260619 09:01:10.815160 2578 raft_consensus.cc:2454] T 442eace1507040779557647f319df0ad P a018d10fc9e547ce9bfba8421bb8770f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c7ed360640b4867afe317a02f50e811 in term 0.
I20260619 09:01:10.815171 2654 raft_consensus.cc:2781] T 442eace1507040779557647f319df0ad P 1c7ed360640b4867afe317a02f50e811 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 09:01:10.815204 2654 raft_consensus.cc:482] T 442eace1507040779557647f319df0ad P 1c7ed360640b4867afe317a02f50e811 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 09:01:10.815222 2654 raft_consensus.cc:3037] T 442eace1507040779557647f319df0ad P 1c7ed360640b4867afe317a02f50e811 [term 0 FOLLOWER]: Advancing to term 1
I20260619 09:01:10.815891 2654 raft_consensus.cc:504] T 442eace1507040779557647f319df0ad P 1c7ed360640b4867afe317a02f50e811 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55a2824214a14a79afa7dccda9503fc5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37703 } } peers { permanent_uuid: "1c7ed360640b4867afe317a02f50e811" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33217 } } peers { permanent_uuid: "a018d10fc9e547ce9bfba8421bb8770f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37317 } }
I20260619 09:01:10.816028 2654 leader_election.cc:288] T 442eace1507040779557647f319df0ad P 1c7ed360640b4867afe317a02f50e811 [CANDIDATE]: Term 1 election: Requested vote from peers 55a2824214a14a79afa7dccda9503fc5 (127.0.98.1:37703), a018d10fc9e547ce9bfba8421bb8770f (127.0.98.3:37317)
I20260619 09:01:10.816157 2298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "442eace1507040779557647f319df0ad" candidate_uuid: "1c7ed360640b4867afe317a02f50e811" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "55a2824214a14a79afa7dccda9503fc5"
I20260619 09:01:10.816226 2298 raft_consensus.cc:3037] T 442eace1507040779557647f319df0ad P 55a2824214a14a79afa7dccda9503fc5 [term 0 FOLLOWER]: Advancing to term 1
I20260619 09:01:10.816226 2578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "442eace1507040779557647f319df0ad" candidate_uuid: "1c7ed360640b4867afe317a02f50e811" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a018d10fc9e547ce9bfba8421bb8770f"
I20260619 09:01:10.816313 2578 raft_consensus.cc:3037] T 442eace1507040779557647f319df0ad P a018d10fc9e547ce9bfba8421bb8770f [term 0 FOLLOWER]: Advancing to term 1
I20260619 09:01:10.816838 2298 raft_consensus.cc:2454] T 442eace1507040779557647f319df0ad P 55a2824214a14a79afa7dccda9503fc5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c7ed360640b4867afe317a02f50e811 in term 1.
I20260619 09:01:10.816946 2578 raft_consensus.cc:2454] T 442eace1507040779557647f319df0ad P a018d10fc9e547ce9bfba8421bb8770f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c7ed360640b4867afe317a02f50e811 in term 1.
I20260619 09:01:10.817003 2369 leader_election.cc:302] T 442eace1507040779557647f319df0ad P 1c7ed360640b4867afe317a02f50e811 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c7ed360640b4867afe317a02f50e811, 55a2824214a14a79afa7dccda9503fc5; no voters:
I20260619 09:01:10.817090 2654 raft_consensus.cc:2781] T 442eace1507040779557647f319df0ad P 1c7ed360640b4867afe317a02f50e811 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 09:01:10.817211 2654 raft_consensus.cc:686] T 442eace1507040779557647f319df0ad P 1c7ed360640b4867afe317a02f50e811 [term 1 LEADER]: Becoming Leader. State: Replica: 1c7ed360640b4867afe317a02f50e811, State: Running, Role: LEADER
I20260619 09:01:10.817313 2654 consensus_queue.cc:237] T 442eace1507040779557647f319df0ad P 1c7ed360640b4867afe317a02f50e811 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55a2824214a14a79afa7dccda9503fc5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37703 } } peers { permanent_uuid: "1c7ed360640b4867afe317a02f50e811" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33217 } } peers { permanent_uuid: "a018d10fc9e547ce9bfba8421bb8770f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37317 } }
I20260619 09:01:10.817741 2671 tablet_replica.cc:439] T 442eace1507040779557647f319df0ad P 1c7ed360640b4867afe317a02f50e811: TxnStatusTablet state changed. Reason: New leader 1c7ed360640b4867afe317a02f50e811. Latest consensus state: current_term: 1 leader_uuid: "1c7ed360640b4867afe317a02f50e811" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55a2824214a14a79afa7dccda9503fc5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37703 } } peers { permanent_uuid: "1c7ed360640b4867afe317a02f50e811" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33217 } } peers { permanent_uuid: "a018d10fc9e547ce9bfba8421bb8770f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37317 } } }
I20260619 09:01:10.817823 2671 tablet_replica.cc:442] T 442eace1507040779557647f319df0ad P 1c7ed360640b4867afe317a02f50e811: This TxnStatusTablet replica's current role is: LEADER
I20260619 09:01:10.818001 2673 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260619 09:01:10.818111 2673 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260619 09:01:10.818066 2140 catalog_manager.cc:5495] T 442eace1507040779557647f319df0ad P 1c7ed360640b4867afe317a02f50e811 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1c7ed360640b4867afe317a02f50e811 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "1c7ed360640b4867afe317a02f50e811" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55a2824214a14a79afa7dccda9503fc5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37703 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c7ed360640b4867afe317a02f50e811" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33217 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a018d10fc9e547ce9bfba8421bb8770f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37317 } health_report { overall_health: UNKNOWN } } }
I20260619 09:01:10.824183 2646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:10.821572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48266 (local address 127.0.98.2:33217)
0619 09:01:10.821611 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:10.821612 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:10.821642 (+ 30us) server_negotiation.cc:407] Connection header received
0619 09:01:10.821729 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:10.821729 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:10.821742 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:10.821798 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 09:01:10.821900 (+ 102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:10.822373 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:10.823454 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:10.823635 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:10.823697 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:10.823883 (+ 186us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 09:01:10.823992 (+ 109us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 09:01:10.824061 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0619 09:01:10.824118 (+ 57us) server_negotiation.cc:299] Negotiation successful
0619 09:01:10.824135 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260619 09:01:10.824239 2665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:10.821519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33217 (local address 127.0.0.1:48266)
0619 09:01:10.821597 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0619 09:01:10.821607 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 09:01:10.821658 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:01:10.821801 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:01:10.821802 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:01:10.821803 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 09:01:10.821847 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:10.821849 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:10.822440 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:10.822441 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:10.823356 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:10.823359 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:10.823704 (+ 345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:10.823705 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:10.823770 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:10.823786 (+ 16us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 09:01:10.824056 (+ 270us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 09:01:10.824061 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 09:01:10.824113 (+ 52us) client_negotiation.cc:240] Negotiation successful
0619 09:01:10.824161 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260619 09:01:10.826013 2578 raft_consensus.cc:1261] T 442eace1507040779557647f319df0ad P a018d10fc9e547ce9bfba8421bb8770f [term 1 FOLLOWER]: Refusing update from remote peer 1c7ed360640b4867afe317a02f50e811: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 09:01:10.826013 2298 raft_consensus.cc:1261] T 442eace1507040779557647f319df0ad P 55a2824214a14a79afa7dccda9503fc5 [term 1 FOLLOWER]: Refusing update from remote peer 1c7ed360640b4867afe317a02f50e811: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 09:01:10.826279 2671 consensus_queue.cc:1035] T 442eace1507040779557647f319df0ad P 1c7ed360640b4867afe317a02f50e811 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a018d10fc9e547ce9bfba8421bb8770f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 09:01:10.826397 2654 consensus_queue.cc:1035] T 442eace1507040779557647f319df0ad P 1c7ed360640b4867afe317a02f50e811 [LEADER]: Connected to new peer: Peer: permanent_uuid: "55a2824214a14a79afa7dccda9503fc5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37703 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 09:01:10.827476 2651 tablet_replica.cc:439] T 442eace1507040779557647f319df0ad P 55a2824214a14a79afa7dccda9503fc5: TxnStatusTablet state changed. Reason: New leader 1c7ed360640b4867afe317a02f50e811. Latest consensus state: current_term: 1 leader_uuid: "1c7ed360640b4867afe317a02f50e811" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55a2824214a14a79afa7dccda9503fc5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37703 } } peers { permanent_uuid: "1c7ed360640b4867afe317a02f50e811" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33217 } } peers { permanent_uuid: "a018d10fc9e547ce9bfba8421bb8770f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37317 } } }
I20260619 09:01:10.827559 2651 tablet_replica.cc:442] T 442eace1507040779557647f319df0ad P 55a2824214a14a79afa7dccda9503fc5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 09:01:10.829794 2655 tablet_replica.cc:439] T 442eace1507040779557647f319df0ad P a018d10fc9e547ce9bfba8421bb8770f: TxnStatusTablet state changed. Reason: New leader 1c7ed360640b4867afe317a02f50e811. Latest consensus state: current_term: 1 leader_uuid: "1c7ed360640b4867afe317a02f50e811" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55a2824214a14a79afa7dccda9503fc5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37703 } } peers { permanent_uuid: "1c7ed360640b4867afe317a02f50e811" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33217 } } peers { permanent_uuid: "a018d10fc9e547ce9bfba8421bb8770f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37317 } } }
I20260619 09:01:10.829895 2655 tablet_replica.cc:442] T 442eace1507040779557647f319df0ad P a018d10fc9e547ce9bfba8421bb8770f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 09:01:10.830483 2654 tablet_replica.cc:439] T 442eace1507040779557647f319df0ad P 1c7ed360640b4867afe317a02f50e811: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1c7ed360640b4867afe317a02f50e811" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55a2824214a14a79afa7dccda9503fc5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37703 } } peers { permanent_uuid: "1c7ed360640b4867afe317a02f50e811" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33217 } } peers { permanent_uuid: "a018d10fc9e547ce9bfba8421bb8770f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37317 } } }
I20260619 09:01:10.830569 2654 tablet_replica.cc:442] T 442eace1507040779557647f319df0ad P 1c7ed360640b4867afe317a02f50e811: This TxnStatusTablet replica's current role is: LEADER
I20260619 09:01:10.830619 2654 tablet_replica.cc:439] T 442eace1507040779557647f319df0ad P 1c7ed360640b4867afe317a02f50e811: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1c7ed360640b4867afe317a02f50e811" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55a2824214a14a79afa7dccda9503fc5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37703 } } peers { permanent_uuid: "1c7ed360640b4867afe317a02f50e811" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33217 } } peers { permanent_uuid: "a018d10fc9e547ce9bfba8421bb8770f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37317 } } }
I20260619 09:01:10.830670 2654 tablet_replica.cc:442] T 442eace1507040779557647f319df0ad P 1c7ed360640b4867afe317a02f50e811: This TxnStatusTablet replica's current role is: LEADER
I20260619 09:01:10.831207 2655 tablet_replica.cc:439] T 442eace1507040779557647f319df0ad P a018d10fc9e547ce9bfba8421bb8770f: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1c7ed360640b4867afe317a02f50e811" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55a2824214a14a79afa7dccda9503fc5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37703 } } peers { permanent_uuid: "1c7ed360640b4867afe317a02f50e811" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33217 } } peers { permanent_uuid: "a018d10fc9e547ce9bfba8421bb8770f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37317 } } }
I20260619 09:01:10.831286 2655 tablet_replica.cc:442] T 442eace1507040779557647f319df0ad P a018d10fc9e547ce9bfba8421bb8770f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 09:01:10.831413 2651 tablet_replica.cc:439] T 442eace1507040779557647f319df0ad P 55a2824214a14a79afa7dccda9503fc5: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1c7ed360640b4867afe317a02f50e811" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55a2824214a14a79afa7dccda9503fc5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37703 } } peers { permanent_uuid: "1c7ed360640b4867afe317a02f50e811" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33217 } } peers { permanent_uuid: "a018d10fc9e547ce9bfba8421bb8770f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37317 } } }
I20260619 09:01:10.831442 2676 mvcc.cc:205] Tried to move back new op lower bound from 7298497211700711424 to 7298497211667894272. Current Snapshot: MvccSnapshot[applied={T|T < 7298497211700711424 or (T in {7298497211700711424})}]
I20260619 09:01:10.831490 2651 tablet_replica.cc:442] T 442eace1507040779557647f319df0ad P 55a2824214a14a79afa7dccda9503fc5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 09:01:10.831526 2677 mvcc.cc:205] Tried to move back new op lower bound from 7298497211700711424 to 7298497211667894272. Current Snapshot: MvccSnapshot[applied={T|T < 7298497211700711424}]
I20260619 09:01:10.835214 2644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:10.832730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41596 (local address 127.0.98.1:37703)
0619 09:01:10.832781 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:10.832783 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:10.832792 (+ 9us) server_negotiation.cc:407] Connection header received
0619 09:01:10.832817 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:10.832818 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:10.832833 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:10.832906 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 09:01:10.832986 (+ 80us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:10.833717 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:10.834412 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:10.834706 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:10.834772 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:10.834914 (+ 142us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 09:01:10.835034 (+ 120us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 09:01:10.835102 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0619 09:01:10.835124 (+ 22us) server_negotiation.cc:299] Negotiation successful
0619 09:01:10.835146 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260619 09:01:10.839684 2646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:10.837461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48805 (local address 127.0.98.2:33217)
0619 09:01:10.837504 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:10.837506 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:10.837613 (+ 107us) server_negotiation.cc:407] Connection header received
0619 09:01:10.837677 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:10.837678 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:10.837687 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:10.837727 (+ 40us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 09:01:10.837835 (+ 108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:10.838281 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:10.839074 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:10.839264 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:10.839314 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:10.839477 (+ 163us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 09:01:10.839509 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 09:01:10.839546 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0619 09:01:10.839595 (+ 49us) server_negotiation.cc:299] Negotiation successful
0619 09:01:10.839615 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260619 09:01:10.839723 2688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:10.837400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33217 (local address 127.0.98.1:48805)
0619 09:01:10.837555 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0619 09:01:10.837565 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 09:01:10.837624 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:01:10.837727 (+ 103us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:01:10.837728 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:01:10.837729 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 09:01:10.837782 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:10.837784 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:10.838345 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:10.838346 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:10.839000 (+ 654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:10.839002 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:10.839318 (+ 316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:10.839319 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:10.839379 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:10.839394 (+ 15us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 09:01:10.839552 (+ 158us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 09:01:10.839554 (+ 2us) client_negotiation.cc:769] Sending connection context
0619 09:01:10.839594 (+ 40us) client_negotiation.cc:240] Negotiation successful
0619 09:01:10.839633 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":79,"threads_started":1}
I20260619 09:01:10.841816 2578 raft_consensus.cc:1261] T 03c76fa0d5864de7bfd9bdebd6a1c11c P a018d10fc9e547ce9bfba8421bb8770f [term 1 FOLLOWER]: Refusing update from remote peer 55a2824214a14a79afa7dccda9503fc5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 09:01:10.841825 2438 raft_consensus.cc:1261] T 03c76fa0d5864de7bfd9bdebd6a1c11c P 1c7ed360640b4867afe317a02f50e811 [term 1 FOLLOWER]: Refusing update from remote peer 55a2824214a14a79afa7dccda9503fc5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 09:01:10.842044 2658 consensus_queue.cc:1035] T 03c76fa0d5864de7bfd9bdebd6a1c11c P 55a2824214a14a79afa7dccda9503fc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a018d10fc9e547ce9bfba8421bb8770f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 09:01:10.842175 2651 consensus_queue.cc:1035] T 03c76fa0d5864de7bfd9bdebd6a1c11c P 55a2824214a14a79afa7dccda9503fc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c7ed360640b4867afe317a02f50e811" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 09:01:10.843743 2679 mvcc.cc:205] Tried to move back new op lower bound from 7298497211765145600 to 7298497211181936640. Current Snapshot: MvccSnapshot[applied={T|T < 7298497211765145600}]
I20260619 09:01:10.850570 2692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:10.847673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37703 (local address 127.0.98.2:60201)
0619 09:01:10.847811 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0619 09:01:10.847823 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 09:01:10.847877 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:01:10.847997 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:01:10.847998 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:01:10.847999 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 09:01:10.848046 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:10.848047 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:10.848881 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:10.848882 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:10.849726 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:10.849728 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:10.850193 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:10.850194 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:10.850260 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:10.850277 (+ 17us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 09:01:10.850435 (+ 158us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 09:01:10.850436 (+ 1us) client_negotiation.cc:769] Sending connection context
0619 09:01:10.850469 (+ 33us) client_negotiation.cc:240] Negotiation successful
0619 09:01:10.850503 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":77,"threads_started":1}
I20260619 09:01:10.850562 2644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:10.847846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60201 (local address 127.0.98.1:37703)
0619 09:01:10.847884 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:10.847885 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:10.847894 (+ 9us) server_negotiation.cc:407] Connection header received
0619 09:01:10.847916 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:10.847917 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:10.847927 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:10.847994 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 09:01:10.848083 (+ 89us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:10.848798 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:10.849790 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:10.850103 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:10.850165 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:10.850350 (+ 185us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 09:01:10.850382 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 09:01:10.850432 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0619 09:01:10.850473 (+ 41us) server_negotiation.cc:299] Negotiation successful
0619 09:01:10.850490 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260619 09:01:10.860080 2139 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36394:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 09:01:10.860191 2139 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36394:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 09:01:10.860899 2139 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5b3e060ff3214f0a95cf22a183e6e061: Sending DeleteTablet for 3 replicas of tablet 03c76fa0d5864de7bfd9bdebd6a1c11c
I20260619 09:01:10.861243 2418 tablet_service.cc:1514] Processing DeleteTablet for tablet 03c76fa0d5864de7bfd9bdebd6a1c11c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 09:01:10 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48254
I20260619 09:01:10.861212 2558 tablet_service.cc:1514] Processing DeleteTablet for tablet 03c76fa0d5864de7bfd9bdebd6a1c11c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 09:01:10 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58520
I20260619 09:01:10.861337 2278 tablet_service.cc:1514] Processing DeleteTablet for tablet 03c76fa0d5864de7bfd9bdebd6a1c11c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 09:01:10 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41594
I20260619 09:01:10.861586 2697 tablet_replica.cc:330] T 03c76fa0d5864de7bfd9bdebd6a1c11c P 55a2824214a14a79afa7dccda9503fc5: stopping tablet replica
I20260619 09:01:10.861781 2697 raft_consensus.cc:2229] T 03c76fa0d5864de7bfd9bdebd6a1c11c P 55a2824214a14a79afa7dccda9503fc5 [term 1 LEADER]: Raft consensus shutting down.
I20260619 09:01:10.861760 2139 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36394:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260619 09:01:10.861881 2697 raft_consensus.cc:2258] T 03c76fa0d5864de7bfd9bdebd6a1c11c P 55a2824214a14a79afa7dccda9503fc5 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260619 09:01:10.861896 2139 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 09:01:10.863178 2558 tablet_service.cc:1467] Processing CreateTablet for tablet d55f3893cb4c481e84dda45709150235 (DEFAULT_TABLE table=test-table [id=59ceb3348fc7405494ecb67ae7f2e945]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 09:01:10.863303 2558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d55f3893cb4c481e84dda45709150235. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 09:01:10.863296 2418 tablet_service.cc:1467] Processing CreateTablet for tablet d55f3893cb4c481e84dda45709150235 (DEFAULT_TABLE table=test-table [id=59ceb3348fc7405494ecb67ae7f2e945]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 09:01:10.863407 2418 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d55f3893cb4c481e84dda45709150235. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 09:01:10.864228 2650 tablet_bootstrap.cc:492] T d55f3893cb4c481e84dda45709150235 P a018d10fc9e547ce9bfba8421bb8770f: Bootstrap starting.
I20260619 09:01:10.864583 2650 tablet_bootstrap.cc:654] T d55f3893cb4c481e84dda45709150235 P a018d10fc9e547ce9bfba8421bb8770f: Neither blocks nor log segments found. Creating new log.
I20260619 09:01:10.864707 2648 tablet_bootstrap.cc:492] T d55f3893cb4c481e84dda45709150235 P 1c7ed360640b4867afe317a02f50e811: Bootstrap starting.
I20260619 09:01:10.865264 2648 tablet_bootstrap.cc:654] T d55f3893cb4c481e84dda45709150235 P 1c7ed360640b4867afe317a02f50e811: Neither blocks nor log segments found. Creating new log.
I20260619 09:01:10.865960 2648 tablet_bootstrap.cc:492] T d55f3893cb4c481e84dda45709150235 P 1c7ed360640b4867afe317a02f50e811: No bootstrap required, opened a new log
I20260619 09:01:10.866024 2648 ts_tablet_manager.cc:1397] T d55f3893cb4c481e84dda45709150235 P 1c7ed360640b4867afe317a02f50e811: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260619 09:01:10.866158 2648 raft_consensus.cc:348] T d55f3893cb4c481e84dda45709150235 P 1c7ed360640b4867afe317a02f50e811 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55a2824214a14a79afa7dccda9503fc5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37703 } } peers { permanent_uuid: "1c7ed360640b4867afe317a02f50e811" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33217 } } peers { permanent_uuid: "a018d10fc9e547ce9bfba8421bb8770f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37317 } }
I20260619 09:01:10.866226 2648 raft_consensus.cc:374] T d55f3893cb4c481e84dda45709150235 P 1c7ed360640b4867afe317a02f50e811 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 09:01:10.866245 2648 raft_consensus.cc:729] T d55f3893cb4c481e84dda45709150235 P 1c7ed360640b4867afe317a02f50e811 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c7ed360640b4867afe317a02f50e811, State: Initialized, Role: FOLLOWER
I20260619 09:01:10.866287 2648 consensus_queue.cc:260] T d55f3893cb4c481e84dda45709150235 P 1c7ed360640b4867afe317a02f50e811 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55a2824214a14a79afa7dccda9503fc5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37703 } } peers { permanent_uuid: "1c7ed360640b4867afe317a02f50e811" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33217 } } peers { permanent_uuid: "a018d10fc9e547ce9bfba8421bb8770f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37317 } }
I20260619 09:01:10.866369 2648 ts_tablet_manager.cc:1428] T d55f3893cb4c481e84dda45709150235 P 1c7ed360640b4867afe317a02f50e811: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260619 09:01:10.863183 2278 tablet_service.cc:1467] Processing CreateTablet for tablet d55f3893cb4c481e84dda45709150235 (DEFAULT_TABLE table=test-table [id=59ceb3348fc7405494ecb67ae7f2e945]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 09:01:10.866667 2278 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d55f3893cb4c481e84dda45709150235. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 09:01:10.867897 2647 tablet_bootstrap.cc:492] T d55f3893cb4c481e84dda45709150235 P 55a2824214a14a79afa7dccda9503fc5: Bootstrap starting.
I20260619 09:01:10.868278 2647 tablet_bootstrap.cc:654] T d55f3893cb4c481e84dda45709150235 P 55a2824214a14a79afa7dccda9503fc5: Neither blocks nor log segments found. Creating new log.
I20260619 09:01:10.862319 2697 ts_tablet_manager.cc:1905] T 03c76fa0d5864de7bfd9bdebd6a1c11c P 55a2824214a14a79afa7dccda9503fc5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 09:01:10.869925 2697 ts_tablet_manager.cc:1918] T 03c76fa0d5864de7bfd9bdebd6a1c11c P 55a2824214a14a79afa7dccda9503fc5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 09:01:10.870005 2697 log.cc:1196] T 03c76fa0d5864de7bfd9bdebd6a1c11c P 55a2824214a14a79afa7dccda9503fc5: Deleting WAL directory at /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781859658134510-392-0/minicluster-data/ts-0/wal/wals/03c76fa0d5864de7bfd9bdebd6a1c11c
I20260619 09:01:10.870244 2697 ts_tablet_manager.cc:1939] T 03c76fa0d5864de7bfd9bdebd6a1c11c P 55a2824214a14a79afa7dccda9503fc5: Deleting consensus metadata
I20260619 09:01:10.870698 2124 catalog_manager.cc:4841] TS 55a2824214a14a79afa7dccda9503fc5 (127.0.98.1:37703): tablet 03c76fa0d5864de7bfd9bdebd6a1c11c (table test-table [id=72e1df6b899b4ec9ac0003e548c91d49]) successfully deleted
I20260619 09:01:10.871140 2696 tablet_replica.cc:330] T 03c76fa0d5864de7bfd9bdebd6a1c11c P 1c7ed360640b4867afe317a02f50e811: stopping tablet replica
I20260619 09:01:10.871239 2650 tablet_bootstrap.cc:492] T d55f3893cb4c481e84dda45709150235 P a018d10fc9e547ce9bfba8421bb8770f: No bootstrap required, opened a new log
I20260619 09:01:10.871266 2695 tablet_replica.cc:330] T 03c76fa0d5864de7bfd9bdebd6a1c11c P a018d10fc9e547ce9bfba8421bb8770f: stopping tablet replica
I20260619 09:01:10.871299 2650 ts_tablet_manager.cc:1397] T d55f3893cb4c481e84dda45709150235 P a018d10fc9e547ce9bfba8421bb8770f: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.000s
I20260619 09:01:10.871490 2695 raft_consensus.cc:2229] T 03c76fa0d5864de7bfd9bdebd6a1c11c P a018d10fc9e547ce9bfba8421bb8770f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 09:01:10.871622 2696 raft_consensus.cc:2229] T 03c76fa0d5864de7bfd9bdebd6a1c11c P 1c7ed360640b4867afe317a02f50e811 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 09:01:10.871690 2696 raft_consensus.cc:2258] T 03c76fa0d5864de7bfd9bdebd6a1c11c P 1c7ed360640b4867afe317a02f50e811 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 09:01:10.871515 2650 raft_consensus.cc:348] T d55f3893cb4c481e84dda45709150235 P a018d10fc9e547ce9bfba8421bb8770f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55a2824214a14a79afa7dccda9503fc5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37703 } } peers { permanent_uuid: "1c7ed360640b4867afe317a02f50e811" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33217 } } peers { permanent_uuid: "a018d10fc9e547ce9bfba8421bb8770f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37317 } }
I20260619 09:01:10.871809 2695 raft_consensus.cc:2258] T 03c76fa0d5864de7bfd9bdebd6a1c11c P a018d10fc9e547ce9bfba8421bb8770f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 09:01:10.871794 2650 raft_consensus.cc:374] T d55f3893cb4c481e84dda45709150235 P a018d10fc9e547ce9bfba8421bb8770f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 09:01:10.871896 2650 raft_consensus.cc:729] T d55f3893cb4c481e84dda45709150235 P a018d10fc9e547ce9bfba8421bb8770f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a018d10fc9e547ce9bfba8421bb8770f, State: Initialized, Role: FOLLOWER
I20260619 09:01:10.871940 2650 consensus_queue.cc:260] T d55f3893cb4c481e84dda45709150235 P a018d10fc9e547ce9bfba8421bb8770f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55a2824214a14a79afa7dccda9503fc5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37703 } } peers { permanent_uuid: "1c7ed360640b4867afe317a02f50e811" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33217 } } peers { permanent_uuid: "a018d10fc9e547ce9bfba8421bb8770f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37317 } }
I20260619 09:01:10.872056 2650 ts_tablet_manager.cc:1428] T d55f3893cb4c481e84dda45709150235 P a018d10fc9e547ce9bfba8421bb8770f: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260619 09:01:10.872198 2647 tablet_bootstrap.cc:492] T d55f3893cb4c481e84dda45709150235 P 55a2824214a14a79afa7dccda9503fc5: No bootstrap required, opened a new log
I20260619 09:01:10.872237 2647 ts_tablet_manager.cc:1397] T d55f3893cb4c481e84dda45709150235 P 55a2824214a14a79afa7dccda9503fc5: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.001s
I20260619 09:01:10.872253 2695 ts_tablet_manager.cc:1905] T 03c76fa0d5864de7bfd9bdebd6a1c11c P a018d10fc9e547ce9bfba8421bb8770f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 09:01:10.872359 2647 raft_consensus.cc:348] T d55f3893cb4c481e84dda45709150235 P 55a2824214a14a79afa7dccda9503fc5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55a2824214a14a79afa7dccda9503fc5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37703 } } peers { permanent_uuid: "1c7ed360640b4867afe317a02f50e811" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33217 } } peers { permanent_uuid: "a018d10fc9e547ce9bfba8421bb8770f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37317 } }
I20260619 09:01:10.872417 2647 raft_consensus.cc:374] T d55f3893cb4c481e84dda45709150235 P 55a2824214a14a79afa7dccda9503fc5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 09:01:10.872435 2647 raft_consensus.cc:729] T d55f3893cb4c481e84dda45709150235 P 55a2824214a14a79afa7dccda9503fc5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55a2824214a14a79afa7dccda9503fc5, State: Initialized, Role: FOLLOWER
I20260619 09:01:10.872475 2647 consensus_queue.cc:260] T d55f3893cb4c481e84dda45709150235 P 55a2824214a14a79afa7dccda9503fc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55a2824214a14a79afa7dccda9503fc5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37703 } } peers { permanent_uuid: "1c7ed360640b4867afe317a02f50e811" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33217 } } peers { permanent_uuid: "a018d10fc9e547ce9bfba8421bb8770f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37317 } }
I20260619 09:01:10.872562 2647 ts_tablet_manager.cc:1428] T d55f3893cb4c481e84dda45709150235 P 55a2824214a14a79afa7dccda9503fc5: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260619 09:01:10.873343 2696 ts_tablet_manager.cc:1905] T 03c76fa0d5864de7bfd9bdebd6a1c11c P 1c7ed360640b4867afe317a02f50e811: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 09:01:10.873679 2695 ts_tablet_manager.cc:1918] T 03c76fa0d5864de7bfd9bdebd6a1c11c P a018d10fc9e547ce9bfba8421bb8770f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 09:01:10.873739 2695 log.cc:1196] T 03c76fa0d5864de7bfd9bdebd6a1c11c P a018d10fc9e547ce9bfba8421bb8770f: Deleting WAL directory at /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781859658134510-392-0/minicluster-data/ts-2/wal/wals/03c76fa0d5864de7bfd9bdebd6a1c11c
I20260619 09:01:10.873939 2695 ts_tablet_manager.cc:1939] T 03c76fa0d5864de7bfd9bdebd6a1c11c P a018d10fc9e547ce9bfba8421bb8770f: Deleting consensus metadata
I20260619 09:01:10.874511 2126 catalog_manager.cc:4841] TS a018d10fc9e547ce9bfba8421bb8770f (127.0.98.3:37317): tablet 03c76fa0d5864de7bfd9bdebd6a1c11c (table test-table [id=72e1df6b899b4ec9ac0003e548c91d49]) successfully deleted
I20260619 09:01:10.875011 2696 ts_tablet_manager.cc:1918] T 03c76fa0d5864de7bfd9bdebd6a1c11c P 1c7ed360640b4867afe317a02f50e811: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 09:01:10.875082 2696 log.cc:1196] T 03c76fa0d5864de7bfd9bdebd6a1c11c P 1c7ed360640b4867afe317a02f50e811: Deleting WAL directory at /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781859658134510-392-0/minicluster-data/ts-1/wal/wals/03c76fa0d5864de7bfd9bdebd6a1c11c
I20260619 09:01:10.875314 2696 ts_tablet_manager.cc:1939] T 03c76fa0d5864de7bfd9bdebd6a1c11c P 1c7ed360640b4867afe317a02f50e811: Deleting consensus metadata
I20260619 09:01:10.875716 2125 catalog_manager.cc:4841] TS 1c7ed360640b4867afe317a02f50e811 (127.0.98.2:33217): tablet 03c76fa0d5864de7bfd9bdebd6a1c11c (table test-table [id=72e1df6b899b4ec9ac0003e548c91d49]) successfully deleted
I20260619 09:01:10.989492 2655 raft_consensus.cc:482] T d55f3893cb4c481e84dda45709150235 P a018d10fc9e547ce9bfba8421bb8770f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 09:01:10.989581 2655 raft_consensus.cc:504] T d55f3893cb4c481e84dda45709150235 P a018d10fc9e547ce9bfba8421bb8770f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55a2824214a14a79afa7dccda9503fc5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37703 } } peers { permanent_uuid: "1c7ed360640b4867afe317a02f50e811" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33217 } } peers { permanent_uuid: "a018d10fc9e547ce9bfba8421bb8770f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37317 } }
I20260619 09:01:10.989863 2655 leader_election.cc:288] T d55f3893cb4c481e84dda45709150235 P a018d10fc9e547ce9bfba8421bb8770f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 55a2824214a14a79afa7dccda9503fc5 (127.0.98.1:37703), 1c7ed360640b4867afe317a02f50e811 (127.0.98.2:33217)
I20260619 09:01:10.993585 2644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:10.990083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45557 (local address 127.0.98.1:37703)
0619 09:01:10.990133 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:10.990135 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:10.990214 (+ 79us) server_negotiation.cc:407] Connection header received
0619 09:01:10.990306 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:10.990307 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:10.990315 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:10.990354 (+ 39us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 09:01:10.990590 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:10.991347 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:10.992906 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:10.993268 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:10.993328 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:10.993340 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0619 09:01:10.993486 (+ 146us) server_negotiation.cc:299] Negotiation successful
0619 09:01:10.993518 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260619 09:01:10.993651 2702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:10.990021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37703 (local address 127.0.98.3:45557)
0619 09:01:10.990146 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0619 09:01:10.990157 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 09:01:10.990237 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:01:10.990391 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:01:10.990392 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:01:10.990398 (+ 6us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 09:01:10.990513 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:10.990514 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:10.991422 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:10.991423 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:10.992789 (+ 1366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:10.992792 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:10.993351 (+ 559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:10.993352 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:10.993417 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:10.993422 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 09:01:10.993467 (+ 45us) client_negotiation.cc:240] Negotiation successful
0619 09:01:10.993512 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"mutex_wait_us":121,"thread_start_us":55,"threads_started":1}
I20260619 09:01:10.993643 2646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:10.990179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48909 (local address 127.0.98.2:33217)
0619 09:01:10.990231 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:10.990233 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:10.990241 (+ 8us) server_negotiation.cc:407] Connection header received
0619 09:01:10.990270 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:10.990270 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:10.990280 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:10.990344 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 09:01:10.990592 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:10.991343 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:10.992889 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:10.993270 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:10.993341 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:10.993352 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0619 09:01:10.993536 (+ 184us) server_negotiation.cc:299] Negotiation successful
0619 09:01:10.993575 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260619 09:01:10.993836 2701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:10.990021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33217 (local address 127.0.98.3:48909)
0619 09:01:10.990146 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0619 09:01:10.990157 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 09:01:10.990214 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:01:10.990391 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:01:10.990392 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:01:10.990400 (+ 8us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 09:01:10.990512 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:10.990514 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:10.991421 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:10.991423 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:10.992789 (+ 1366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:10.992792 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:10.993352 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:10.993354 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:10.993438 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:10.993441 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 09:01:10.993519 (+ 78us) client_negotiation.cc:240] Negotiation successful
0619 09:01:10.993563 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"mutex_wait_us":58,"thread_start_us":68,"threads_started":1}
I20260619 09:01:10.993862 2298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d55f3893cb4c481e84dda45709150235" candidate_uuid: "a018d10fc9e547ce9bfba8421bb8770f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "55a2824214a14a79afa7dccda9503fc5" is_pre_election: true
I20260619 09:01:10.993952 2298 raft_consensus.cc:2454] T d55f3893cb4c481e84dda45709150235 P 55a2824214a14a79afa7dccda9503fc5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a018d10fc9e547ce9bfba8421bb8770f in term 0.
I20260619 09:01:10.994040 2438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d55f3893cb4c481e84dda45709150235" candidate_uuid: "a018d10fc9e547ce9bfba8421bb8770f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c7ed360640b4867afe317a02f50e811" is_pre_election: true
I20260619 09:01:10.994108 2438 raft_consensus.cc:2454] T d55f3893cb4c481e84dda45709150235 P 1c7ed360640b4867afe317a02f50e811 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a018d10fc9e547ce9bfba8421bb8770f in term 0.
I20260619 09:01:10.994596 2509 leader_election.cc:302] T d55f3893cb4c481e84dda45709150235 P a018d10fc9e547ce9bfba8421bb8770f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 55a2824214a14a79afa7dccda9503fc5, a018d10fc9e547ce9bfba8421bb8770f; no voters:
I20260619 09:01:10.994721 2655 raft_consensus.cc:2781] T d55f3893cb4c481e84dda45709150235 P a018d10fc9e547ce9bfba8421bb8770f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 09:01:10.994757 2655 raft_consensus.cc:482] T d55f3893cb4c481e84dda45709150235 P a018d10fc9e547ce9bfba8421bb8770f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 09:01:10.994777 2655 raft_consensus.cc:3037] T d55f3893cb4c481e84dda45709150235 P a018d10fc9e547ce9bfba8421bb8770f [term 0 FOLLOWER]: Advancing to term 1
I20260619 09:01:10.995414 2655 raft_consensus.cc:504] T d55f3893cb4c481e84dda45709150235 P a018d10fc9e547ce9bfba8421bb8770f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55a2824214a14a79afa7dccda9503fc5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37703 } } peers { permanent_uuid: "1c7ed360640b4867afe317a02f50e811" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33217 } } peers { permanent_uuid: "a018d10fc9e547ce9bfba8421bb8770f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37317 } }
I20260619 09:01:10.995537 2655 leader_election.cc:288] T d55f3893cb4c481e84dda45709150235 P a018d10fc9e547ce9bfba8421bb8770f [CANDIDATE]: Term 1 election: Requested vote from peers 55a2824214a14a79afa7dccda9503fc5 (127.0.98.1:37703), 1c7ed360640b4867afe317a02f50e811 (127.0.98.2:33217)
I20260619 09:01:10.995684 2438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d55f3893cb4c481e84dda45709150235" candidate_uuid: "a018d10fc9e547ce9bfba8421bb8770f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c7ed360640b4867afe317a02f50e811"
I20260619 09:01:10.995682 2298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d55f3893cb4c481e84dda45709150235" candidate_uuid: "a018d10fc9e547ce9bfba8421bb8770f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "55a2824214a14a79afa7dccda9503fc5"
I20260619 09:01:10.995764 2298 raft_consensus.cc:3037] T d55f3893cb4c481e84dda45709150235 P 55a2824214a14a79afa7dccda9503fc5 [term 0 FOLLOWER]: Advancing to term 1
I20260619 09:01:10.995762 2438 raft_consensus.cc:3037] T d55f3893cb4c481e84dda45709150235 P 1c7ed360640b4867afe317a02f50e811 [term 0 FOLLOWER]: Advancing to term 1
I20260619 09:01:10.996466 2298 raft_consensus.cc:2454] T d55f3893cb4c481e84dda45709150235 P 55a2824214a14a79afa7dccda9503fc5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a018d10fc9e547ce9bfba8421bb8770f in term 1.
I20260619 09:01:10.996474 2438 raft_consensus.cc:2454] T d55f3893cb4c481e84dda45709150235 P 1c7ed360640b4867afe317a02f50e811 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a018d10fc9e547ce9bfba8421bb8770f in term 1.
I20260619 09:01:10.996626 2510 leader_election.cc:302] T d55f3893cb4c481e84dda45709150235 P a018d10fc9e547ce9bfba8421bb8770f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c7ed360640b4867afe317a02f50e811, a018d10fc9e547ce9bfba8421bb8770f; no voters:
I20260619 09:01:10.996723 2655 raft_consensus.cc:2781] T d55f3893cb4c481e84dda45709150235 P a018d10fc9e547ce9bfba8421bb8770f [term 1 FOLLOWER]: Leader election won for term 1
I20260619 09:01:10.996815 2655 raft_consensus.cc:686] T d55f3893cb4c481e84dda45709150235 P a018d10fc9e547ce9bfba8421bb8770f [term 1 LEADER]: Becoming Leader. State: Replica: a018d10fc9e547ce9bfba8421bb8770f, State: Running, Role: LEADER
I20260619 09:01:10.996868 2655 consensus_queue.cc:237] T d55f3893cb4c481e84dda45709150235 P a018d10fc9e547ce9bfba8421bb8770f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55a2824214a14a79afa7dccda9503fc5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37703 } } peers { permanent_uuid: "1c7ed360640b4867afe317a02f50e811" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33217 } } peers { permanent_uuid: "a018d10fc9e547ce9bfba8421bb8770f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37317 } }
I20260619 09:01:10.997502 2139 catalog_manager.cc:5495] T d55f3893cb4c481e84dda45709150235 P a018d10fc9e547ce9bfba8421bb8770f reported cstate change: term changed from 0 to 1, leader changed from <none> to a018d10fc9e547ce9bfba8421bb8770f (127.0.98.3). New cstate: current_term: 1 leader_uuid: "a018d10fc9e547ce9bfba8421bb8770f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55a2824214a14a79afa7dccda9503fc5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37703 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c7ed360640b4867afe317a02f50e811" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33217 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a018d10fc9e547ce9bfba8421bb8770f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37317 } health_report { overall_health: HEALTHY } } }
I20260619 09:01:11.014159 2645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:11.012096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58528 (local address 127.0.98.3:37317)
0619 09:01:11.012152 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:11.012154 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:11.012162 (+ 8us) server_negotiation.cc:407] Connection header received
0619 09:01:11.012194 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:11.012195 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:11.012211 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:11.012266 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 09:01:11.012341 (+ 75us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:11.012793 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:11.013512 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:11.013712 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:11.013759 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:11.013863 (+ 104us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 09:01:11.013961 (+ 98us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 09:01:11.014010 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0619 09:01:11.014037 (+ 27us) server_negotiation.cc:299] Negotiation successful
0619 09:01:11.014051 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260619 09:01:11.018657 2646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:11.016723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41589 (local address 127.0.98.2:33217)
0619 09:01:11.016792 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:11.016794 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:11.016804 (+ 10us) server_negotiation.cc:407] Connection header received
0619 09:01:11.016849 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:11.016850 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:11.016859 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:11.016908 (+ 49us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 09:01:11.017004 (+ 96us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:11.017440 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:11.018115 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:11.018293 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:11.018345 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:11.018485 (+ 140us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 09:01:11.018509 (+ 24us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 09:01:11.018542 (+ 33us) server_negotiation.cc:1035] Waiting for connection context
0619 09:01:11.018588 (+ 46us) server_negotiation.cc:299] Negotiation successful
0619 09:01:11.018603 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 09:01:11.018877 2707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:11.016628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33217 (local address 127.0.98.3:41589)
0619 09:01:11.016757 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0619 09:01:11.016767 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 09:01:11.016809 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:01:11.016917 (+ 108us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:01:11.016917 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:01:11.016918 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 09:01:11.016968 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:11.016969 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:11.017489 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:11.017489 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:11.018045 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:11.018047 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:11.018361 (+ 314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:11.018361 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:11.018413 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:11.018425 (+ 12us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 09:01:11.018556 (+ 131us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 09:01:11.018557 (+ 1us) client_negotiation.cc:769] Sending connection context
0619 09:01:11.018586 (+ 29us) client_negotiation.cc:240] Negotiation successful
0619 09:01:11.018616 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":67,"threads_started":1}
I20260619 09:01:11.020685 2298 raft_consensus.cc:1261] T d55f3893cb4c481e84dda45709150235 P 55a2824214a14a79afa7dccda9503fc5 [term 1 FOLLOWER]: Refusing update from remote peer a018d10fc9e547ce9bfba8421bb8770f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 09:01:11.020738 2438 raft_consensus.cc:1261] T d55f3893cb4c481e84dda45709150235 P 1c7ed360640b4867afe317a02f50e811 [term 1 FOLLOWER]: Refusing update from remote peer a018d10fc9e547ce9bfba8421bb8770f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 09:01:11.020916 2655 consensus_queue.cc:1035] T d55f3893cb4c481e84dda45709150235 P a018d10fc9e547ce9bfba8421bb8770f [LEADER]: Connected to new peer: Peer: permanent_uuid: "55a2824214a14a79afa7dccda9503fc5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37703 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 09:01:11.021059 2703 consensus_queue.cc:1035] T d55f3893cb4c481e84dda45709150235 P a018d10fc9e547ce9bfba8421bb8770f [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c7ed360640b4867afe317a02f50e811" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 09:01:11.027218 2645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:11.025230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51687 (local address 127.0.98.3:37317)
0619 09:01:11.025280 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:11.025282 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:11.025289 (+ 7us) server_negotiation.cc:407] Connection header received
0619 09:01:11.025307 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:11.025307 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:11.025316 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:11.025386 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 09:01:11.025482 (+ 96us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:11.025936 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:11.026614 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:11.026806 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:11.026885 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:11.026999 (+ 114us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 09:01:11.027029 (+ 30us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 09:01:11.027074 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0619 09:01:11.027136 (+ 62us) server_negotiation.cc:299] Negotiation successful
0619 09:01:11.027153 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260619 09:01:11.027236 2692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:11.025080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37317 (local address 127.0.98.2:51687)
0619 09:01:11.025119 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0619 09:01:11.025130 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 09:01:11.025165 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:01:11.025391 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:01:11.025392 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:01:11.025393 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 09:01:11.025431 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:11.025433 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:11.025997 (+ 564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:11.025998 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:11.026539 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:11.026541 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:11.026885 (+ 344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:11.026885 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:11.026937 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:11.026943 (+ 6us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 09:01:11.027088 (+ 145us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 09:01:11.027089 (+ 1us) client_negotiation.cc:769] Sending connection context
0619 09:01:11.027132 (+ 43us) client_negotiation.cc:240] Negotiation successful
0619 09:01:11.027172 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19}
W20260619 09:01:11.027464 2679 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260619 09:01:12.548692 2140 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36394:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 09:01:12.548784 2140 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36394:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 09:01:12.549603 2140 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5b3e060ff3214f0a95cf22a183e6e061: Sending DeleteTablet for 3 replicas of tablet d55f3893cb4c481e84dda45709150235
I20260619 09:01:12.549825 2278 tablet_service.cc:1514] Processing DeleteTablet for tablet d55f3893cb4c481e84dda45709150235 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 09:01:12 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41594
I20260619 09:01:12.549861 2418 tablet_service.cc:1514] Processing DeleteTablet for tablet d55f3893cb4c481e84dda45709150235 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 09:01:12 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48254
I20260619 09:01:12.549860 2558 tablet_service.cc:1514] Processing DeleteTablet for tablet d55f3893cb4c481e84dda45709150235 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 09:01:12 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58520
I20260619 09:01:12.550043 2736 tablet_replica.cc:330] T d55f3893cb4c481e84dda45709150235 P 55a2824214a14a79afa7dccda9503fc5: stopping tablet replica
I20260619 09:01:12.550050 2738 tablet_replica.cc:330] T d55f3893cb4c481e84dda45709150235 P a018d10fc9e547ce9bfba8421bb8770f: stopping tablet replica
I20260619 09:01:12.550141 2736 raft_consensus.cc:2229] T d55f3893cb4c481e84dda45709150235 P 55a2824214a14a79afa7dccda9503fc5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 09:01:12.550159 2738 raft_consensus.cc:2229] T d55f3893cb4c481e84dda45709150235 P a018d10fc9e547ce9bfba8421bb8770f [term 1 LEADER]: Raft consensus shutting down.
I20260619 09:01:12.550215 2736 raft_consensus.cc:2258] T d55f3893cb4c481e84dda45709150235 P 55a2824214a14a79afa7dccda9503fc5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 09:01:12.550239 2738 raft_consensus.cc:2258] T d55f3893cb4c481e84dda45709150235 P a018d10fc9e547ce9bfba8421bb8770f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 09:01:12.550417 2737 tablet_replica.cc:330] T d55f3893cb4c481e84dda45709150235 P 1c7ed360640b4867afe317a02f50e811: stopping tablet replica
I20260619 09:01:12.550477 2738 ts_tablet_manager.cc:1905] T d55f3893cb4c481e84dda45709150235 P a018d10fc9e547ce9bfba8421bb8770f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 09:01:12.550480 2736 ts_tablet_manager.cc:1905] T d55f3893cb4c481e84dda45709150235 P 55a2824214a14a79afa7dccda9503fc5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 09:01:12.550537 2737 raft_consensus.cc:2229] T d55f3893cb4c481e84dda45709150235 P 1c7ed360640b4867afe317a02f50e811 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 09:01:12.550580 2737 raft_consensus.cc:2258] T d55f3893cb4c481e84dda45709150235 P 1c7ed360640b4867afe317a02f50e811 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 09:01:12.550881 2737 ts_tablet_manager.cc:1905] T d55f3893cb4c481e84dda45709150235 P 1c7ed360640b4867afe317a02f50e811: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 09:01:12.552069 2738 ts_tablet_manager.cc:1918] T d55f3893cb4c481e84dda45709150235 P a018d10fc9e547ce9bfba8421bb8770f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 09:01:12.552069 2736 ts_tablet_manager.cc:1918] T d55f3893cb4c481e84dda45709150235 P 55a2824214a14a79afa7dccda9503fc5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 09:01:12.552124 2736 log.cc:1196] T d55f3893cb4c481e84dda45709150235 P 55a2824214a14a79afa7dccda9503fc5: Deleting WAL directory at /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781859658134510-392-0/minicluster-data/ts-0/wal/wals/d55f3893cb4c481e84dda45709150235
I20260619 09:01:12.552229 2738 log.cc:1196] T d55f3893cb4c481e84dda45709150235 P a018d10fc9e547ce9bfba8421bb8770f: Deleting WAL directory at /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781859658134510-392-0/minicluster-data/ts-2/wal/wals/d55f3893cb4c481e84dda45709150235
I20260619 09:01:12.552330 2736 ts_tablet_manager.cc:1939] T d55f3893cb4c481e84dda45709150235 P 55a2824214a14a79afa7dccda9503fc5: Deleting consensus metadata
I20260619 09:01:12.552521 2737 ts_tablet_manager.cc:1918] T d55f3893cb4c481e84dda45709150235 P 1c7ed360640b4867afe317a02f50e811: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 09:01:12.552549 2738 ts_tablet_manager.cc:1939] T d55f3893cb4c481e84dda45709150235 P a018d10fc9e547ce9bfba8421bb8770f: Deleting consensus metadata
I20260619 09:01:12.552628 2737 log.cc:1196] T d55f3893cb4c481e84dda45709150235 P 1c7ed360640b4867afe317a02f50e811: Deleting WAL directory at /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781859658134510-392-0/minicluster-data/ts-1/wal/wals/d55f3893cb4c481e84dda45709150235
I20260619 09:01:12.552732 2124 catalog_manager.cc:4841] TS 55a2824214a14a79afa7dccda9503fc5 (127.0.98.1:37703): tablet d55f3893cb4c481e84dda45709150235 (table test-table [id=59ceb3348fc7405494ecb67ae7f2e945]) successfully deleted
I20260619 09:01:12.552891 2737 ts_tablet_manager.cc:1939] T d55f3893cb4c481e84dda45709150235 P 1c7ed360640b4867afe317a02f50e811: Deleting consensus metadata
I20260619 09:01:12.553076 2126 catalog_manager.cc:4841] TS a018d10fc9e547ce9bfba8421bb8770f (127.0.98.3:37317): tablet d55f3893cb4c481e84dda45709150235 (table test-table [id=59ceb3348fc7405494ecb67ae7f2e945]) successfully deleted
I20260619 09:01:12.553207 2125 catalog_manager.cc:4841] TS 1c7ed360640b4867afe317a02f50e811 (127.0.98.2:33217): tablet d55f3893cb4c481e84dda45709150235 (table test-table [id=59ceb3348fc7405494ecb67ae7f2e945]) successfully deleted
Jun 19 09:01:12 dist-test-slave-mtdr krb5kdc[2095](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859672, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 09:01:12 dist-test-slave-mtdr krb5kdc[2095](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859672, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 09:01:12.573036 2746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:12.566513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46644 (local address 127.0.98.62:46507)
0619 09:01:12.566681 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:12.566683 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:12.566691 (+ 8us) server_negotiation.cc:407] Connection header received
0619 09:01:12.566716 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:12.566717 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:12.566726 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:12.566788 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:01:12.567397 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:12.567902 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:12.568644 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:12.568845 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:12.568909 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:12.571012 (+ 2103us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:01:12.571013 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:01:12.571013 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:01:12.571019 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:01:12.572209 (+ 1190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:01:12.572504 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:01:12.572505 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:01:12.572505 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:01:12.572549 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:01:12.572670 (+ 121us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:01:12.572671 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:01:12.572671 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:01:12.572760 (+ 89us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:01:12.572828 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0619 09:01:12.572898 (+ 70us) server_negotiation.cc:299] Negotiation successful
0619 09:01:12.572953 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":93,"threads_started":1}
I20260619 09:01:12.574246 2140 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46644:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 09:01:12.574397 2140 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 09:01:12.575690 2558 tablet_service.cc:1467] Processing CreateTablet for tablet 37684ff032624c5e9f145ecaa4c33866 (DEFAULT_TABLE table=test-table [id=9d2aae80d796443da942a7eeadf63d0c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 09:01:12.575686 2418 tablet_service.cc:1467] Processing CreateTablet for tablet 37684ff032624c5e9f145ecaa4c33866 (DEFAULT_TABLE table=test-table [id=9d2aae80d796443da942a7eeadf63d0c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 09:01:12.575805 2418 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37684ff032624c5e9f145ecaa4c33866. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 09:01:12.575807 2558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37684ff032624c5e9f145ecaa4c33866. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 09:01:12.576766 2747 tablet_bootstrap.cc:492] T 37684ff032624c5e9f145ecaa4c33866 P a018d10fc9e547ce9bfba8421bb8770f: Bootstrap starting.
I20260619 09:01:12.576813 2748 tablet_bootstrap.cc:492] T 37684ff032624c5e9f145ecaa4c33866 P 1c7ed360640b4867afe317a02f50e811: Bootstrap starting.
I20260619 09:01:12.576835 2278 tablet_service.cc:1467] Processing CreateTablet for tablet 37684ff032624c5e9f145ecaa4c33866 (DEFAULT_TABLE table=test-table [id=9d2aae80d796443da942a7eeadf63d0c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 09:01:12.576941 2278 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37684ff032624c5e9f145ecaa4c33866. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 09:01:12.577152 2747 tablet_bootstrap.cc:654] T 37684ff032624c5e9f145ecaa4c33866 P a018d10fc9e547ce9bfba8421bb8770f: Neither blocks nor log segments found. Creating new log.
I20260619 09:01:12.577194 2748 tablet_bootstrap.cc:654] T 37684ff032624c5e9f145ecaa4c33866 P 1c7ed360640b4867afe317a02f50e811: Neither blocks nor log segments found. Creating new log.
I20260619 09:01:12.577713 2747 tablet_bootstrap.cc:492] T 37684ff032624c5e9f145ecaa4c33866 P a018d10fc9e547ce9bfba8421bb8770f: No bootstrap required, opened a new log
I20260619 09:01:12.577756 2747 ts_tablet_manager.cc:1397] T 37684ff032624c5e9f145ecaa4c33866 P a018d10fc9e547ce9bfba8421bb8770f: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260619 09:01:12.577867 2747 raft_consensus.cc:348] T 37684ff032624c5e9f145ecaa4c33866 P a018d10fc9e547ce9bfba8421bb8770f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55a2824214a14a79afa7dccda9503fc5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37703 } } peers { permanent_uuid: "1c7ed360640b4867afe317a02f50e811" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33217 } } peers { permanent_uuid: "a018d10fc9e547ce9bfba8421bb8770f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37317 } }
I20260619 09:01:12.577925 2747 raft_consensus.cc:374] T 37684ff032624c5e9f145ecaa4c33866 P a018d10fc9e547ce9bfba8421bb8770f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 09:01:12.577942 2747 raft_consensus.cc:729] T 37684ff032624c5e9f145ecaa4c33866 P a018d10fc9e547ce9bfba8421bb8770f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a018d10fc9e547ce9bfba8421bb8770f, State: Initialized, Role: FOLLOWER
I20260619 09:01:12.577992 2747 consensus_queue.cc:260] T 37684ff032624c5e9f145ecaa4c33866 P a018d10fc9e547ce9bfba8421bb8770f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55a2824214a14a79afa7dccda9503fc5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37703 } } peers { permanent_uuid: "1c7ed360640b4867afe317a02f50e811" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33217 } } peers { permanent_uuid: "a018d10fc9e547ce9bfba8421bb8770f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37317 } }
I20260619 09:01:12.578096 2747 ts_tablet_manager.cc:1428] T 37684ff032624c5e9f145ecaa4c33866 P a018d10fc9e547ce9bfba8421bb8770f: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260619 09:01:12.578214 2751 tablet_bootstrap.cc:492] T 37684ff032624c5e9f145ecaa4c33866 P 55a2824214a14a79afa7dccda9503fc5: Bootstrap starting.
I20260619 09:01:12.578289 2748 tablet_bootstrap.cc:492] T 37684ff032624c5e9f145ecaa4c33866 P 1c7ed360640b4867afe317a02f50e811: No bootstrap required, opened a new log
I20260619 09:01:12.578347 2748 ts_tablet_manager.cc:1397] T 37684ff032624c5e9f145ecaa4c33866 P 1c7ed360640b4867afe317a02f50e811: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 09:01:12.578480 2748 raft_consensus.cc:348] T 37684ff032624c5e9f145ecaa4c33866 P 1c7ed360640b4867afe317a02f50e811 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55a2824214a14a79afa7dccda9503fc5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37703 } } peers { permanent_uuid: "1c7ed360640b4867afe317a02f50e811" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33217 } } peers { permanent_uuid: "a018d10fc9e547ce9bfba8421bb8770f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37317 } }
I20260619 09:01:12.578567 2748 raft_consensus.cc:374] T 37684ff032624c5e9f145ecaa4c33866 P 1c7ed360640b4867afe317a02f50e811 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 09:01:12.578596 2748 raft_consensus.cc:729] T 37684ff032624c5e9f145ecaa4c33866 P 1c7ed360640b4867afe317a02f50e811 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c7ed360640b4867afe317a02f50e811, State: Initialized, Role: FOLLOWER
I20260619 09:01:12.578595 2751 tablet_bootstrap.cc:654] T 37684ff032624c5e9f145ecaa4c33866 P 55a2824214a14a79afa7dccda9503fc5: Neither blocks nor log segments found. Creating new log.
I20260619 09:01:12.578635 2748 consensus_queue.cc:260] T 37684ff032624c5e9f145ecaa4c33866 P 1c7ed360640b4867afe317a02f50e811 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55a2824214a14a79afa7dccda9503fc5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37703 } } peers { permanent_uuid: "1c7ed360640b4867afe317a02f50e811" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33217 } } peers { permanent_uuid: "a018d10fc9e547ce9bfba8421bb8770f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37317 } }
I20260619 09:01:12.578728 2748 ts_tablet_manager.cc:1428] T 37684ff032624c5e9f145ecaa4c33866 P 1c7ed360640b4867afe317a02f50e811: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260619 09:01:12.579350 2751 tablet_bootstrap.cc:492] T 37684ff032624c5e9f145ecaa4c33866 P 55a2824214a14a79afa7dccda9503fc5: No bootstrap required, opened a new log
I20260619 09:01:12.579402 2751 ts_tablet_manager.cc:1397] T 37684ff032624c5e9f145ecaa4c33866 P 55a2824214a14a79afa7dccda9503fc5: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260619 09:01:12.579504 2751 raft_consensus.cc:348] T 37684ff032624c5e9f145ecaa4c33866 P 55a2824214a14a79afa7dccda9503fc5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55a2824214a14a79afa7dccda9503fc5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37703 } } peers { permanent_uuid: "1c7ed360640b4867afe317a02f50e811" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33217 } } peers { permanent_uuid: "a018d10fc9e547ce9bfba8421bb8770f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37317 } }
I20260619 09:01:12.579566 2751 raft_consensus.cc:374] T 37684ff032624c5e9f145ecaa4c33866 P 55a2824214a14a79afa7dccda9503fc5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 09:01:12.579578 2751 raft_consensus.cc:729] T 37684ff032624c5e9f145ecaa4c33866 P 55a2824214a14a79afa7dccda9503fc5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55a2824214a14a79afa7dccda9503fc5, State: Initialized, Role: FOLLOWER
I20260619 09:01:12.579613 2751 consensus_queue.cc:260] T 37684ff032624c5e9f145ecaa4c33866 P 55a2824214a14a79afa7dccda9503fc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55a2824214a14a79afa7dccda9503fc5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37703 } } peers { permanent_uuid: "1c7ed360640b4867afe317a02f50e811" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33217 } } peers { permanent_uuid: "a018d10fc9e547ce9bfba8421bb8770f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37317 } }
I20260619 09:01:12.579756 2751 ts_tablet_manager.cc:1428] T 37684ff032624c5e9f145ecaa4c33866 P 55a2824214a14a79afa7dccda9503fc5: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260619 09:01:12.639129 2722 raft_consensus.cc:482] T 37684ff032624c5e9f145ecaa4c33866 P 1c7ed360640b4867afe317a02f50e811 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 09:01:12.639220 2722 raft_consensus.cc:504] T 37684ff032624c5e9f145ecaa4c33866 P 1c7ed360640b4867afe317a02f50e811 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55a2824214a14a79afa7dccda9503fc5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37703 } } peers { permanent_uuid: "1c7ed360640b4867afe317a02f50e811" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33217 } } peers { permanent_uuid: "a018d10fc9e547ce9bfba8421bb8770f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37317 } }
I20260619 09:01:12.639374 2722 leader_election.cc:288] T 37684ff032624c5e9f145ecaa4c33866 P 1c7ed360640b4867afe317a02f50e811 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 55a2824214a14a79afa7dccda9503fc5 (127.0.98.1:37703), a018d10fc9e547ce9bfba8421bb8770f (127.0.98.3:37317)
I20260619 09:01:12.639554 2298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37684ff032624c5e9f145ecaa4c33866" candidate_uuid: "1c7ed360640b4867afe317a02f50e811" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "55a2824214a14a79afa7dccda9503fc5" is_pre_election: true
I20260619 09:01:12.639585 2578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37684ff032624c5e9f145ecaa4c33866" candidate_uuid: "1c7ed360640b4867afe317a02f50e811" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a018d10fc9e547ce9bfba8421bb8770f" is_pre_election: true
I20260619 09:01:12.639665 2298 raft_consensus.cc:2454] T 37684ff032624c5e9f145ecaa4c33866 P 55a2824214a14a79afa7dccda9503fc5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c7ed360640b4867afe317a02f50e811 in term 0.
I20260619 09:01:12.639679 2578 raft_consensus.cc:2454] T 37684ff032624c5e9f145ecaa4c33866 P a018d10fc9e547ce9bfba8421bb8770f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c7ed360640b4867afe317a02f50e811 in term 0.
I20260619 09:01:12.639824 2369 leader_election.cc:302] T 37684ff032624c5e9f145ecaa4c33866 P 1c7ed360640b4867afe317a02f50e811 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c7ed360640b4867afe317a02f50e811, 55a2824214a14a79afa7dccda9503fc5; no voters:
I20260619 09:01:12.639930 2722 raft_consensus.cc:2781] T 37684ff032624c5e9f145ecaa4c33866 P 1c7ed360640b4867afe317a02f50e811 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 09:01:12.639971 2722 raft_consensus.cc:482] T 37684ff032624c5e9f145ecaa4c33866 P 1c7ed360640b4867afe317a02f50e811 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 09:01:12.639990 2722 raft_consensus.cc:3037] T 37684ff032624c5e9f145ecaa4c33866 P 1c7ed360640b4867afe317a02f50e811 [term 0 FOLLOWER]: Advancing to term 1
I20260619 09:01:12.640627 2722 raft_consensus.cc:504] T 37684ff032624c5e9f145ecaa4c33866 P 1c7ed360640b4867afe317a02f50e811 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55a2824214a14a79afa7dccda9503fc5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37703 } } peers { permanent_uuid: "1c7ed360640b4867afe317a02f50e811" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33217 } } peers { permanent_uuid: "a018d10fc9e547ce9bfba8421bb8770f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37317 } }
I20260619 09:01:12.640735 2722 leader_election.cc:288] T 37684ff032624c5e9f145ecaa4c33866 P 1c7ed360640b4867afe317a02f50e811 [CANDIDATE]: Term 1 election: Requested vote from peers 55a2824214a14a79afa7dccda9503fc5 (127.0.98.1:37703), a018d10fc9e547ce9bfba8421bb8770f (127.0.98.3:37317)
I20260619 09:01:12.640883 2298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37684ff032624c5e9f145ecaa4c33866" candidate_uuid: "1c7ed360640b4867afe317a02f50e811" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "55a2824214a14a79afa7dccda9503fc5"
I20260619 09:01:12.640884 2578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37684ff032624c5e9f145ecaa4c33866" candidate_uuid: "1c7ed360640b4867afe317a02f50e811" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a018d10fc9e547ce9bfba8421bb8770f"
I20260619 09:01:12.640951 2578 raft_consensus.cc:3037] T 37684ff032624c5e9f145ecaa4c33866 P a018d10fc9e547ce9bfba8421bb8770f [term 0 FOLLOWER]: Advancing to term 1
I20260619 09:01:12.640951 2298 raft_consensus.cc:3037] T 37684ff032624c5e9f145ecaa4c33866 P 55a2824214a14a79afa7dccda9503fc5 [term 0 FOLLOWER]: Advancing to term 1
I20260619 09:01:12.641496 2578 raft_consensus.cc:2454] T 37684ff032624c5e9f145ecaa4c33866 P a018d10fc9e547ce9bfba8421bb8770f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c7ed360640b4867afe317a02f50e811 in term 1.
I20260619 09:01:12.641503 2298 raft_consensus.cc:2454] T 37684ff032624c5e9f145ecaa4c33866 P 55a2824214a14a79afa7dccda9503fc5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c7ed360640b4867afe317a02f50e811 in term 1.
I20260619 09:01:12.641649 2369 leader_election.cc:302] T 37684ff032624c5e9f145ecaa4c33866 P 1c7ed360640b4867afe317a02f50e811 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c7ed360640b4867afe317a02f50e811, 55a2824214a14a79afa7dccda9503fc5; no voters:
I20260619 09:01:12.641737 2722 raft_consensus.cc:2781] T 37684ff032624c5e9f145ecaa4c33866 P 1c7ed360640b4867afe317a02f50e811 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 09:01:12.641786 2722 raft_consensus.cc:686] T 37684ff032624c5e9f145ecaa4c33866 P 1c7ed360640b4867afe317a02f50e811 [term 1 LEADER]: Becoming Leader. State: Replica: 1c7ed360640b4867afe317a02f50e811, State: Running, Role: LEADER
I20260619 09:01:12.641834 2722 consensus_queue.cc:237] T 37684ff032624c5e9f145ecaa4c33866 P 1c7ed360640b4867afe317a02f50e811 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55a2824214a14a79afa7dccda9503fc5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37703 } } peers { permanent_uuid: "1c7ed360640b4867afe317a02f50e811" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33217 } } peers { permanent_uuid: "a018d10fc9e547ce9bfba8421bb8770f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37317 } }
I20260619 09:01:12.642388 2138 catalog_manager.cc:5495] T 37684ff032624c5e9f145ecaa4c33866 P 1c7ed360640b4867afe317a02f50e811 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1c7ed360640b4867afe317a02f50e811 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "1c7ed360640b4867afe317a02f50e811" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55a2824214a14a79afa7dccda9503fc5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37703 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c7ed360640b4867afe317a02f50e811" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33217 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a018d10fc9e547ce9bfba8421bb8770f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37317 } health_report { overall_health: UNKNOWN } } }
I20260619 09:01:12.653548 2756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:12.651456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44286 (local address 127.0.98.2:33217)
0619 09:01:12.651575 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:12.651576 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:12.651584 (+ 8us) server_negotiation.cc:407] Connection header received
0619 09:01:12.651605 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:12.651605 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:12.651615 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:12.651670 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 09:01:12.651783 (+ 113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:12.652258 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:12.652952 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:12.653218 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:12.653271 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:12.653383 (+ 112us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 09:01:12.653410 (+ 27us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 09:01:12.653442 (+ 32us) server_negotiation.cc:1035] Waiting for connection context
0619 09:01:12.653475 (+ 33us) server_negotiation.cc:299] Negotiation successful
0619 09:01:12.653493 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95,"thread_start_us":67,"threads_started":1}
I20260619 09:01:12.657282 2756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:12.655416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40697 (local address 127.0.98.2:33217)
0619 09:01:12.655447 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:12.655449 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:12.655459 (+ 10us) server_negotiation.cc:407] Connection header received
0619 09:01:12.655520 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:12.655521 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:12.655530 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:12.655571 (+ 41us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 09:01:12.655659 (+ 88us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:12.656100 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:12.656755 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:12.656921 (+ 166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:12.656969 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:12.657077 (+ 108us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 09:01:12.657103 (+ 26us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 09:01:12.657141 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0619 09:01:12.657187 (+ 46us) server_negotiation.cc:299] Negotiation successful
0619 09:01:12.657206 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260619 09:01:12.657298 2759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:12.655306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33217 (local address 127.0.98.2:40697)
0619 09:01:12.655417 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0619 09:01:12.655427 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 09:01:12.655470 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:01:12.655581 (+ 111us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:01:12.655582 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:01:12.655583 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 09:01:12.655618 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:12.655619 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:12.656161 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:12.656161 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:12.656692 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:12.656693 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:12.656971 (+ 278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:12.656972 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:12.657018 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:12.657021 (+ 3us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 09:01:12.657147 (+ 126us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 09:01:12.657148 (+ 1us) client_negotiation.cc:769] Sending connection context
0619 09:01:12.657187 (+ 39us) client_negotiation.cc:240] Negotiation successful
0619 09:01:12.657221 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":89,"thread_start_us":55,"threads_started":1}
I20260619 09:01:12.658893 2578 raft_consensus.cc:1261] T 37684ff032624c5e9f145ecaa4c33866 P a018d10fc9e547ce9bfba8421bb8770f [term 1 FOLLOWER]: Refusing update from remote peer 1c7ed360640b4867afe317a02f50e811: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 09:01:12.658919 2298 raft_consensus.cc:1261] T 37684ff032624c5e9f145ecaa4c33866 P 55a2824214a14a79afa7dccda9503fc5 [term 1 FOLLOWER]: Refusing update from remote peer 1c7ed360640b4867afe317a02f50e811: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 09:01:12.659090 2713 consensus_queue.cc:1035] T 37684ff032624c5e9f145ecaa4c33866 P 1c7ed360640b4867afe317a02f50e811 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a018d10fc9e547ce9bfba8421bb8770f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 09:01:12.659206 2717 consensus_queue.cc:1035] T 37684ff032624c5e9f145ecaa4c33866 P 1c7ed360640b4867afe317a02f50e811 [LEADER]: Connected to new peer: Peer: permanent_uuid: "55a2824214a14a79afa7dccda9503fc5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37703 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 09:01:12.668145 2138 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46644:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 09:01:12.668236 2138 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46644:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 09:01:12.669448 2138 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5b3e060ff3214f0a95cf22a183e6e061: Sending DeleteTablet for 3 replicas of tablet 37684ff032624c5e9f145ecaa4c33866
I20260619 09:01:12.669709 2418 tablet_service.cc:1514] Processing DeleteTablet for tablet 37684ff032624c5e9f145ecaa4c33866 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 09:01:12 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48254
I20260619 09:01:12.669705 2558 tablet_service.cc:1514] Processing DeleteTablet for tablet 37684ff032624c5e9f145ecaa4c33866 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 09:01:12 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58520
I20260619 09:01:12.669781 2737 tablet_replica.cc:330] T 37684ff032624c5e9f145ecaa4c33866 P 1c7ed360640b4867afe317a02f50e811: stopping tablet replica
I20260619 09:01:12.669795 2738 tablet_replica.cc:330] T 37684ff032624c5e9f145ecaa4c33866 P a018d10fc9e547ce9bfba8421bb8770f: stopping tablet replica
I20260619 09:01:12.669826 2737 raft_consensus.cc:2229] T 37684ff032624c5e9f145ecaa4c33866 P 1c7ed360640b4867afe317a02f50e811 [term 1 LEADER]: Raft consensus shutting down.
I20260619 09:01:12.669854 2738 raft_consensus.cc:2229] T 37684ff032624c5e9f145ecaa4c33866 P a018d10fc9e547ce9bfba8421bb8770f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 09:01:12.669886 2738 raft_consensus.cc:2258] T 37684ff032624c5e9f145ecaa4c33866 P a018d10fc9e547ce9bfba8421bb8770f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 09:01:12.669898 2737 raft_consensus.cc:2258] T 37684ff032624c5e9f145ecaa4c33866 P 1c7ed360640b4867afe317a02f50e811 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 09:01:12.669912 2138 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46644:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 09:01:12.670029 2138 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 09:01:12.670163 2737 ts_tablet_manager.cc:1905] T 37684ff032624c5e9f145ecaa4c33866 P 1c7ed360640b4867afe317a02f50e811: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 09:01:12.671077 2558 tablet_service.cc:1467] Processing CreateTablet for tablet 124960d55e8c4759aec3e246ee71cb99 (DEFAULT_TABLE table=test-table [id=d24f6ab0e36a418385f061ec3090dcbc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 09:01:12.671178 2558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 124960d55e8c4759aec3e246ee71cb99. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 09:01:12.671751 2737 ts_tablet_manager.cc:1918] T 37684ff032624c5e9f145ecaa4c33866 P 1c7ed360640b4867afe317a02f50e811: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 09:01:12.671800 2737 log.cc:1196] T 37684ff032624c5e9f145ecaa4c33866 P 1c7ed360640b4867afe317a02f50e811: Deleting WAL directory at /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781859658134510-392-0/minicluster-data/ts-1/wal/wals/37684ff032624c5e9f145ecaa4c33866
I20260619 09:01:12.672004 2737 ts_tablet_manager.cc:1939] T 37684ff032624c5e9f145ecaa4c33866 P 1c7ed360640b4867afe317a02f50e811: Deleting consensus metadata
I20260619 09:01:12.672120 2418 tablet_service.cc:1467] Processing CreateTablet for tablet 124960d55e8c4759aec3e246ee71cb99 (DEFAULT_TABLE table=test-table [id=d24f6ab0e36a418385f061ec3090dcbc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 09:01:12.672223 2418 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 124960d55e8c4759aec3e246ee71cb99. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 09:01:12.672387 2747 tablet_bootstrap.cc:492] T 124960d55e8c4759aec3e246ee71cb99 P a018d10fc9e547ce9bfba8421bb8770f: Bootstrap starting.
I20260619 09:01:12.672700 2747 tablet_bootstrap.cc:654] T 124960d55e8c4759aec3e246ee71cb99 P a018d10fc9e547ce9bfba8421bb8770f: Neither blocks nor log segments found. Creating new log.
I20260619 09:01:12.673139 2748 tablet_bootstrap.cc:492] T 124960d55e8c4759aec3e246ee71cb99 P 1c7ed360640b4867afe317a02f50e811: Bootstrap starting.
I20260619 09:01:12.673225 2125 catalog_manager.cc:4841] TS 1c7ed360640b4867afe317a02f50e811 (127.0.98.2:33217): tablet 37684ff032624c5e9f145ecaa4c33866 (table test-table [id=9d2aae80d796443da942a7eeadf63d0c]) successfully deleted
I20260619 09:01:12.673346 2278 tablet_service.cc:1514] Processing DeleteTablet for tablet 37684ff032624c5e9f145ecaa4c33866 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 09:01:12 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41594
I20260619 09:01:12.673372 2277 tablet_service.cc:1467] Processing CreateTablet for tablet 124960d55e8c4759aec3e246ee71cb99 (DEFAULT_TABLE table=test-table [id=d24f6ab0e36a418385f061ec3090dcbc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 09:01:12.673444 2748 tablet_bootstrap.cc:654] T 124960d55e8c4759aec3e246ee71cb99 P 1c7ed360640b4867afe317a02f50e811: Neither blocks nor log segments found. Creating new log.
I20260619 09:01:12.673457 2277 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 124960d55e8c4759aec3e246ee71cb99. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 09:01:12.673574 2738 ts_tablet_manager.cc:1905] T 37684ff032624c5e9f145ecaa4c33866 P a018d10fc9e547ce9bfba8421bb8770f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 09:01:12.673767 2736 tablet_replica.cc:330] T 37684ff032624c5e9f145ecaa4c33866 P 55a2824214a14a79afa7dccda9503fc5: stopping tablet replica
I20260619 09:01:12.673826 2736 raft_consensus.cc:2229] T 37684ff032624c5e9f145ecaa4c33866 P 55a2824214a14a79afa7dccda9503fc5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 09:01:12.673864 2736 raft_consensus.cc:2258] T 37684ff032624c5e9f145ecaa4c33866 P 55a2824214a14a79afa7dccda9503fc5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 09:01:12.674203 2747 tablet_bootstrap.cc:492] T 124960d55e8c4759aec3e246ee71cb99 P a018d10fc9e547ce9bfba8421bb8770f: No bootstrap required, opened a new log
I20260619 09:01:12.674248 2747 ts_tablet_manager.cc:1397] T 124960d55e8c4759aec3e246ee71cb99 P a018d10fc9e547ce9bfba8421bb8770f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 09:01:12.674343 2747 raft_consensus.cc:348] T 124960d55e8c4759aec3e246ee71cb99 P a018d10fc9e547ce9bfba8421bb8770f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55a2824214a14a79afa7dccda9503fc5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37703 } } peers { permanent_uuid: "1c7ed360640b4867afe317a02f50e811" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33217 } } peers { permanent_uuid: "a018d10fc9e547ce9bfba8421bb8770f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37317 } }
I20260619 09:01:12.674397 2747 raft_consensus.cc:374] T 124960d55e8c4759aec3e246ee71cb99 P a018d10fc9e547ce9bfba8421bb8770f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 09:01:12.674415 2747 raft_consensus.cc:729] T 124960d55e8c4759aec3e246ee71cb99 P a018d10fc9e547ce9bfba8421bb8770f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a018d10fc9e547ce9bfba8421bb8770f, State: Initialized, Role: FOLLOWER
I20260619 09:01:12.674451 2747 consensus_queue.cc:260] T 124960d55e8c4759aec3e246ee71cb99 P a018d10fc9e547ce9bfba8421bb8770f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55a2824214a14a79afa7dccda9503fc5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37703 } } peers { permanent_uuid: "1c7ed360640b4867afe317a02f50e811" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33217 } } peers { permanent_uuid: "a018d10fc9e547ce9bfba8421bb8770f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37317 } }
I20260619 09:01:12.674528 2747 ts_tablet_manager.cc:1428] T 124960d55e8c4759aec3e246ee71cb99 P a018d10fc9e547ce9bfba8421bb8770f: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260619 09:01:12.674750 2751 tablet_bootstrap.cc:492] T 124960d55e8c4759aec3e246ee71cb99 P 55a2824214a14a79afa7dccda9503fc5: Bootstrap starting.
I20260619 09:01:12.674757 2738 ts_tablet_manager.cc:1918] T 37684ff032624c5e9f145ecaa4c33866 P a018d10fc9e547ce9bfba8421bb8770f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 09:01:12.674799 2738 log.cc:1196] T 37684ff032624c5e9f145ecaa4c33866 P a018d10fc9e547ce9bfba8421bb8770f: Deleting WAL directory at /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781859658134510-392-0/minicluster-data/ts-2/wal/wals/37684ff032624c5e9f145ecaa4c33866
I20260619 09:01:12.675004 2738 ts_tablet_manager.cc:1939] T 37684ff032624c5e9f145ecaa4c33866 P a018d10fc9e547ce9bfba8421bb8770f: Deleting consensus metadata
I20260619 09:01:12.675050 2751 tablet_bootstrap.cc:654] T 124960d55e8c4759aec3e246ee71cb99 P 55a2824214a14a79afa7dccda9503fc5: Neither blocks nor log segments found. Creating new log.
I20260619 09:01:12.675266 2126 catalog_manager.cc:4841] TS a018d10fc9e547ce9bfba8421bb8770f (127.0.98.3:37317): tablet 37684ff032624c5e9f145ecaa4c33866 (table test-table [id=9d2aae80d796443da942a7eeadf63d0c]) successfully deleted
I20260619 09:01:12.675758 2751 tablet_bootstrap.cc:492] T 124960d55e8c4759aec3e246ee71cb99 P 55a2824214a14a79afa7dccda9503fc5: No bootstrap required, opened a new log
I20260619 09:01:12.675807 2751 ts_tablet_manager.cc:1397] T 124960d55e8c4759aec3e246ee71cb99 P 55a2824214a14a79afa7dccda9503fc5: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260619 09:01:12.675899 2751 raft_consensus.cc:348] T 124960d55e8c4759aec3e246ee71cb99 P 55a2824214a14a79afa7dccda9503fc5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55a2824214a14a79afa7dccda9503fc5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37703 } } peers { permanent_uuid: "1c7ed360640b4867afe317a02f50e811" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33217 } } peers { permanent_uuid: "a018d10fc9e547ce9bfba8421bb8770f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37317 } }
I20260619 09:01:12.675951 2751 raft_consensus.cc:374] T 124960d55e8c4759aec3e246ee71cb99 P 55a2824214a14a79afa7dccda9503fc5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 09:01:12.675969 2751 raft_consensus.cc:729] T 124960d55e8c4759aec3e246ee71cb99 P 55a2824214a14a79afa7dccda9503fc5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55a2824214a14a79afa7dccda9503fc5, State: Initialized, Role: FOLLOWER
I20260619 09:01:12.676012 2751 consensus_queue.cc:260] T 124960d55e8c4759aec3e246ee71cb99 P 55a2824214a14a79afa7dccda9503fc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55a2824214a14a79afa7dccda9503fc5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37703 } } peers { permanent_uuid: "1c7ed360640b4867afe317a02f50e811" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33217 } } peers { permanent_uuid: "a018d10fc9e547ce9bfba8421bb8770f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37317 } }
I20260619 09:01:12.676090 2751 ts_tablet_manager.cc:1428] T 124960d55e8c4759aec3e246ee71cb99 P 55a2824214a14a79afa7dccda9503fc5: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260619 09:01:12.676470 2736 ts_tablet_manager.cc:1905] T 37684ff032624c5e9f145ecaa4c33866 P 55a2824214a14a79afa7dccda9503fc5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 09:01:12.676803 2748 tablet_bootstrap.cc:492] T 124960d55e8c4759aec3e246ee71cb99 P 1c7ed360640b4867afe317a02f50e811: No bootstrap required, opened a new log
I20260619 09:01:12.676843 2748 ts_tablet_manager.cc:1397] T 124960d55e8c4759aec3e246ee71cb99 P 1c7ed360640b4867afe317a02f50e811: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260619 09:01:12.676925 2748 raft_consensus.cc:348] T 124960d55e8c4759aec3e246ee71cb99 P 1c7ed360640b4867afe317a02f50e811 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55a2824214a14a79afa7dccda9503fc5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37703 } } peers { permanent_uuid: "1c7ed360640b4867afe317a02f50e811" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33217 } } peers { permanent_uuid: "a018d10fc9e547ce9bfba8421bb8770f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37317 } }
I20260619 09:01:12.676977 2748 raft_consensus.cc:374] T 124960d55e8c4759aec3e246ee71cb99 P 1c7ed360640b4867afe317a02f50e811 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 09:01:12.676995 2748 raft_consensus.cc:729] T 124960d55e8c4759aec3e246ee71cb99 P 1c7ed360640b4867afe317a02f50e811 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c7ed360640b4867afe317a02f50e811, State: Initialized, Role: FOLLOWER
I20260619 09:01:12.677034 2748 consensus_queue.cc:260] T 124960d55e8c4759aec3e246ee71cb99 P 1c7ed360640b4867afe317a02f50e811 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55a2824214a14a79afa7dccda9503fc5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37703 } } peers { permanent_uuid: "1c7ed360640b4867afe317a02f50e811" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33217 } } peers { permanent_uuid: "a018d10fc9e547ce9bfba8421bb8770f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37317 } }
I20260619 09:01:12.677105 2748 ts_tablet_manager.cc:1428] T 124960d55e8c4759aec3e246ee71cb99 P 1c7ed360640b4867afe317a02f50e811: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260619 09:01:12.677816 2736 ts_tablet_manager.cc:1918] T 37684ff032624c5e9f145ecaa4c33866 P 55a2824214a14a79afa7dccda9503fc5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 09:01:12.677860 2736 log.cc:1196] T 37684ff032624c5e9f145ecaa4c33866 P 55a2824214a14a79afa7dccda9503fc5: Deleting WAL directory at /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781859658134510-392-0/minicluster-data/ts-0/wal/wals/37684ff032624c5e9f145ecaa4c33866
I20260619 09:01:12.678059 2736 ts_tablet_manager.cc:1939] T 37684ff032624c5e9f145ecaa4c33866 P 55a2824214a14a79afa7dccda9503fc5: Deleting consensus metadata
I20260619 09:01:12.678331 2124 catalog_manager.cc:4841] TS 55a2824214a14a79afa7dccda9503fc5 (127.0.98.1:37703): tablet 37684ff032624c5e9f145ecaa4c33866 (table test-table [id=9d2aae80d796443da942a7eeadf63d0c]) successfully deleted
I20260619 09:01:12.690676 2727 raft_consensus.cc:482] T 124960d55e8c4759aec3e246ee71cb99 P a018d10fc9e547ce9bfba8421bb8770f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 09:01:12.690778 2727 raft_consensus.cc:504] T 124960d55e8c4759aec3e246ee71cb99 P a018d10fc9e547ce9bfba8421bb8770f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55a2824214a14a79afa7dccda9503fc5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37703 } } peers { permanent_uuid: "1c7ed360640b4867afe317a02f50e811" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33217 } } peers { permanent_uuid: "a018d10fc9e547ce9bfba8421bb8770f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37317 } }
I20260619 09:01:12.690995 2727 leader_election.cc:288] T 124960d55e8c4759aec3e246ee71cb99 P a018d10fc9e547ce9bfba8421bb8770f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 55a2824214a14a79afa7dccda9503fc5 (127.0.98.1:37703), 1c7ed360640b4867afe317a02f50e811 (127.0.98.2:33217)
I20260619 09:01:12.691139 2438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "124960d55e8c4759aec3e246ee71cb99" candidate_uuid: "a018d10fc9e547ce9bfba8421bb8770f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c7ed360640b4867afe317a02f50e811" is_pre_election: true
I20260619 09:01:12.691182 2298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "124960d55e8c4759aec3e246ee71cb99" candidate_uuid: "a018d10fc9e547ce9bfba8421bb8770f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "55a2824214a14a79afa7dccda9503fc5" is_pre_election: true
I20260619 09:01:12.691233 2438 raft_consensus.cc:2454] T 124960d55e8c4759aec3e246ee71cb99 P 1c7ed360640b4867afe317a02f50e811 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a018d10fc9e547ce9bfba8421bb8770f in term 0.
I20260619 09:01:12.691262 2298 raft_consensus.cc:2454] T 124960d55e8c4759aec3e246ee71cb99 P 55a2824214a14a79afa7dccda9503fc5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a018d10fc9e547ce9bfba8421bb8770f in term 0.
I20260619 09:01:12.691378 2510 leader_election.cc:302] T 124960d55e8c4759aec3e246ee71cb99 P a018d10fc9e547ce9bfba8421bb8770f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c7ed360640b4867afe317a02f50e811, a018d10fc9e547ce9bfba8421bb8770f; no voters:
I20260619 09:01:12.691478 2727 raft_consensus.cc:2781] T 124960d55e8c4759aec3e246ee71cb99 P a018d10fc9e547ce9bfba8421bb8770f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 09:01:12.691522 2727 raft_consensus.cc:482] T 124960d55e8c4759aec3e246ee71cb99 P a018d10fc9e547ce9bfba8421bb8770f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 09:01:12.691542 2727 raft_consensus.cc:3037] T 124960d55e8c4759aec3e246ee71cb99 P a018d10fc9e547ce9bfba8421bb8770f [term 0 FOLLOWER]: Advancing to term 1
I20260619 09:01:12.692363 2727 raft_consensus.cc:504] T 124960d55e8c4759aec3e246ee71cb99 P a018d10fc9e547ce9bfba8421bb8770f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55a2824214a14a79afa7dccda9503fc5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37703 } } peers { permanent_uuid: "1c7ed360640b4867afe317a02f50e811" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33217 } } peers { permanent_uuid: "a018d10fc9e547ce9bfba8421bb8770f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37317 } }
I20260619 09:01:12.692497 2727 leader_election.cc:288] T 124960d55e8c4759aec3e246ee71cb99 P a018d10fc9e547ce9bfba8421bb8770f [CANDIDATE]: Term 1 election: Requested vote from peers 55a2824214a14a79afa7dccda9503fc5 (127.0.98.1:37703), 1c7ed360640b4867afe317a02f50e811 (127.0.98.2:33217)
I20260619 09:01:12.692656 2298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "124960d55e8c4759aec3e246ee71cb99" candidate_uuid: "a018d10fc9e547ce9bfba8421bb8770f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "55a2824214a14a79afa7dccda9503fc5"
I20260619 09:01:12.692668 2438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "124960d55e8c4759aec3e246ee71cb99" candidate_uuid: "a018d10fc9e547ce9bfba8421bb8770f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c7ed360640b4867afe317a02f50e811"
I20260619 09:01:12.692732 2438 raft_consensus.cc:3037] T 124960d55e8c4759aec3e246ee71cb99 P 1c7ed360640b4867afe317a02f50e811 [term 0 FOLLOWER]: Advancing to term 1
I20260619 09:01:12.692732 2298 raft_consensus.cc:3037] T 124960d55e8c4759aec3e246ee71cb99 P 55a2824214a14a79afa7dccda9503fc5 [term 0 FOLLOWER]: Advancing to term 1
I20260619 09:01:12.693434 2298 raft_consensus.cc:2454] T 124960d55e8c4759aec3e246ee71cb99 P 55a2824214a14a79afa7dccda9503fc5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a018d10fc9e547ce9bfba8421bb8770f in term 1.
I20260619 09:01:12.693437 2438 raft_consensus.cc:2454] T 124960d55e8c4759aec3e246ee71cb99 P 1c7ed360640b4867afe317a02f50e811 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a018d10fc9e547ce9bfba8421bb8770f in term 1.
I20260619 09:01:12.693593 2509 leader_election.cc:302] T 124960d55e8c4759aec3e246ee71cb99 P a018d10fc9e547ce9bfba8421bb8770f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 55a2824214a14a79afa7dccda9503fc5, a018d10fc9e547ce9bfba8421bb8770f; no voters:
I20260619 09:01:12.693676 2727 raft_consensus.cc:2781] T 124960d55e8c4759aec3e246ee71cb99 P a018d10fc9e547ce9bfba8421bb8770f [term 1 FOLLOWER]: Leader election won for term 1
I20260619 09:01:12.693719 2727 raft_consensus.cc:686] T 124960d55e8c4759aec3e246ee71cb99 P a018d10fc9e547ce9bfba8421bb8770f [term 1 LEADER]: Becoming Leader. State: Replica: a018d10fc9e547ce9bfba8421bb8770f, State: Running, Role: LEADER
I20260619 09:01:12.693764 2727 consensus_queue.cc:237] T 124960d55e8c4759aec3e246ee71cb99 P a018d10fc9e547ce9bfba8421bb8770f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55a2824214a14a79afa7dccda9503fc5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37703 } } peers { permanent_uuid: "1c7ed360640b4867afe317a02f50e811" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33217 } } peers { permanent_uuid: "a018d10fc9e547ce9bfba8421bb8770f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37317 } }
I20260619 09:01:12.694235 2138 catalog_manager.cc:5495] T 124960d55e8c4759aec3e246ee71cb99 P a018d10fc9e547ce9bfba8421bb8770f reported cstate change: term changed from 0 to 1, leader changed from <none> to a018d10fc9e547ce9bfba8421bb8770f (127.0.98.3). New cstate: current_term: 1 leader_uuid: "a018d10fc9e547ce9bfba8421bb8770f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55a2824214a14a79afa7dccda9503fc5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37703 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c7ed360640b4867afe317a02f50e811" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33217 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a018d10fc9e547ce9bfba8421bb8770f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37317 } health_report { overall_health: HEALTHY } } }
I20260619 09:01:12.701730 2767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:12.699470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41612 (local address 127.0.98.3:37317)
0619 09:01:12.699597 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:12.699599 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:12.699606 (+ 7us) server_negotiation.cc:407] Connection header received
0619 09:01:12.699626 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:12.699626 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:12.699634 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:12.699691 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 09:01:12.699805 (+ 114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:12.700291 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:12.701061 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:12.701276 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:12.701329 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:12.701462 (+ 133us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 09:01:12.701496 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 09:01:12.701535 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0619 09:01:12.701612 (+ 77us) server_negotiation.cc:299] Negotiation successful
0619 09:01:12.701631 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":75,"threads_started":1}
I20260619 09:01:12.703540 2438 raft_consensus.cc:1261] T 124960d55e8c4759aec3e246ee71cb99 P 1c7ed360640b4867afe317a02f50e811 [term 1 FOLLOWER]: Refusing update from remote peer a018d10fc9e547ce9bfba8421bb8770f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 09:01:12.703649 2298 raft_consensus.cc:1261] T 124960d55e8c4759aec3e246ee71cb99 P 55a2824214a14a79afa7dccda9503fc5 [term 1 FOLLOWER]: Refusing update from remote peer a018d10fc9e547ce9bfba8421bb8770f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 09:01:12.703747 2730 consensus_queue.cc:1035] T 124960d55e8c4759aec3e246ee71cb99 P a018d10fc9e547ce9bfba8421bb8770f [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c7ed360640b4867afe317a02f50e811" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 09:01:12.703838 2727 consensus_queue.cc:1035] T 124960d55e8c4759aec3e246ee71cb99 P a018d10fc9e547ce9bfba8421bb8770f [LEADER]: Connected to new peer: Peer: permanent_uuid: "55a2824214a14a79afa7dccda9503fc5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37703 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260619 09:01:12.708261 2719 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260619 09:01:14.224037 2138 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46644:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 09:01:14.224141 2138 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46644:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 09:01:14.224869 2138 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5b3e060ff3214f0a95cf22a183e6e061: Sending DeleteTablet for 3 replicas of tablet 124960d55e8c4759aec3e246ee71cb99
I20260619 09:01:14.225107 2418 tablet_service.cc:1514] Processing DeleteTablet for tablet 124960d55e8c4759aec3e246ee71cb99 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 09:01:14 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48254
I20260619 09:01:14.225098 2277 tablet_service.cc:1514] Processing DeleteTablet for tablet 124960d55e8c4759aec3e246ee71cb99 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 09:01:14 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41594
I20260619 09:01:14.225111 2558 tablet_service.cc:1514] Processing DeleteTablet for tablet 124960d55e8c4759aec3e246ee71cb99 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 09:01:14 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58520
I20260619 09:01:14.225294 2794 tablet_replica.cc:330] T 124960d55e8c4759aec3e246ee71cb99 P 1c7ed360640b4867afe317a02f50e811: stopping tablet replica
I20260619 09:01:14.225348 2794 raft_consensus.cc:2229] T 124960d55e8c4759aec3e246ee71cb99 P 1c7ed360640b4867afe317a02f50e811 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 09:01:14.225376 2794 raft_consensus.cc:2258] T 124960d55e8c4759aec3e246ee71cb99 P 1c7ed360640b4867afe317a02f50e811 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 09:01:14.225384 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ez5NB/build/release/bin/kudu with pid 2213
I20260619 09:01:14.226119 2794 ts_tablet_manager.cc:1905] T 124960d55e8c4759aec3e246ee71cb99 P 1c7ed360640b4867afe317a02f50e811: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 09:01:14.227358 2795 tablet_replica.cc:330] T 124960d55e8c4759aec3e246ee71cb99 P a018d10fc9e547ce9bfba8421bb8770f: stopping tablet replica
I20260619 09:01:14.227430 2794 ts_tablet_manager.cc:1918] T 124960d55e8c4759aec3e246ee71cb99 P 1c7ed360640b4867afe317a02f50e811: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 09:01:14.227496 2794 log.cc:1196] T 124960d55e8c4759aec3e246ee71cb99 P 1c7ed360640b4867afe317a02f50e811: Deleting WAL directory at /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781859658134510-392-0/minicluster-data/ts-1/wal/wals/124960d55e8c4759aec3e246ee71cb99
I20260619 09:01:14.227523 2795 raft_consensus.cc:2229] T 124960d55e8c4759aec3e246ee71cb99 P a018d10fc9e547ce9bfba8421bb8770f [term 1 LEADER]: Raft consensus shutting down.
I20260619 09:01:14.227639 2795 raft_consensus.cc:2258] T 124960d55e8c4759aec3e246ee71cb99 P a018d10fc9e547ce9bfba8421bb8770f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 09:01:14.227720 2794 ts_tablet_manager.cc:1939] T 124960d55e8c4759aec3e246ee71cb99 P 1c7ed360640b4867afe317a02f50e811: Deleting consensus metadata
I20260619 09:01:14.227914 2795 ts_tablet_manager.cc:1905] T 124960d55e8c4759aec3e246ee71cb99 P a018d10fc9e547ce9bfba8421bb8770f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 09:01:14.228039 2125 catalog_manager.cc:4841] TS 1c7ed360640b4867afe317a02f50e811 (127.0.98.2:33217): tablet 124960d55e8c4759aec3e246ee71cb99 (table test-table [id=d24f6ab0e36a418385f061ec3090dcbc]) successfully deleted
I20260619 09:01:14.229082 2795 ts_tablet_manager.cc:1918] T 124960d55e8c4759aec3e246ee71cb99 P a018d10fc9e547ce9bfba8421bb8770f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 09:01:14.229131 2795 log.cc:1196] T 124960d55e8c4759aec3e246ee71cb99 P a018d10fc9e547ce9bfba8421bb8770f: Deleting WAL directory at /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781859658134510-392-0/minicluster-data/ts-2/wal/wals/124960d55e8c4759aec3e246ee71cb99
I20260619 09:01:14.229316 2795 ts_tablet_manager.cc:1939] T 124960d55e8c4759aec3e246ee71cb99 P a018d10fc9e547ce9bfba8421bb8770f: Deleting consensus metadata
I20260619 09:01:14.229596 2126 catalog_manager.cc:4841] TS a018d10fc9e547ce9bfba8421bb8770f (127.0.98.3:37317): tablet 124960d55e8c4759aec3e246ee71cb99 (table test-table [id=d24f6ab0e36a418385f061ec3090dcbc]) successfully deleted
W20260619 09:01:14.231400 2124 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:37703 (error 108)
I20260619 09:01:14.231856 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ez5NB/build/release/bin/kudu with pid 2353
I20260619 09:01:14.231937 2797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:14.231751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37703 (local address 127.0.0.1:43266)
0619 09:01:14.231856 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0619 09:01:14.231901 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81,"thread_start_us":58,"threads_started":1}
W20260619 09:01:14.232447 2124 catalog_manager.cc:4584] TS 55a2824214a14a79afa7dccda9503fc5 (127.0.98.1:37703): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 124960d55e8c4759aec3e246ee71cb99: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37703: connect: Connection refused (error 111)
I20260619 09:01:14.237691 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ez5NB/build/release/bin/kudu with pid 2493
I20260619 09:01:14.242311 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ez5NB/build/release/bin/kudu with pid 2108
2026-06-19T09:01:14Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (5589 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781859658134510-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 09:01:14 dist-test-slave-mtdr krb5kdc[2800](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 09:01:14 dist-test-slave-mtdr krb5kdc[2800](info): set up 2 sockets
Jun 19 09:01:14 dist-test-slave-mtdr krb5kdc[2800](info): commencing operation
krb5kdc: starting...
W20260619 09:01:15.190258 392 mini_kdc.cc:121] Time spent starting KDC: real 0.929s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781859658134510-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781859658134510-392-0/krb5kdc/test-user.keytab.
Jun 19 09:01:15 dist-test-slave-mtdr krb5kdc[2800](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859675, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T09:01:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T09:01:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781859658134510-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781859658134510-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 09:01:15.298087 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ez5NB/build/release/bin/kudu
/tmp/dist-test-task4ez5NB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781859658134510-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781859658134510-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781859658134510-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781859658134510-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33817
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781859658134510-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33817
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781859658134510-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781859658134510-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781859658134510-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 09:01:15.374572 2813 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 09:01:15.374756 2813 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 09:01:15.374776 2813 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 09:01:15.375918 2813 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 09:01:15.375959 2813 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 09:01:15.375972 2813 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 09:01:15.375983 2813 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 09:01:15.375994 2813 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 09:01:15.377439 2813 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781859658134510-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781859658134510-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33817
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781859658134510-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33817
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781859658134510-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2813
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781859658134510-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 09:01:15.377658 2813 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 09:01:15.377861 2813 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 09:01:15.382266 2819 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:01:15.382323 2821 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:01:15.382364 2813 server_base.cc:1034] running on GCE node
W20260619 09:01:15.382266 2818 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:01:15.382619 2813 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 09:01:15.382928 2813 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 09:01:15.384099 2813 hybrid_clock.cc:648] HybridClock initialized: now 1781859675384083 us; error 30 us; skew 500 ppm
Jun 19 09:01:15 dist-test-slave-mtdr krb5kdc[2800](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859675, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 09:01:15.391129 2813 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 09:01:15.391538 2813 webserver.cc:460] Webserver started at http://127.0.98.62:34811/ using document root <none> and password file <none>
I20260619 09:01:15.391760 2813 fs_manager.cc:359] Metadata directory not provided
I20260619 09:01:15.391808 2813 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 09:01:15.391929 2813 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 09:01:15.392937 2813 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781859658134510-392-0/minicluster-data/master-0/data/instance:
uuid: "85e0fa583d4e4a7b86764d5ab97d067b"
format_stamp: "Formatted at 2026-06-19 09:01:15 on dist-test-slave-mtdr"
server_key: "1c0c6a09b8f4657b310fbb9d0177f1fe"
server_key_iv: "4cc45bf5571d226c9f907433f2273f42"
server_key_version: "encryptionkey@0"
I20260619 09:01:15.393257 2813 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781859658134510-392-0/minicluster-data/master-0/wal/instance:
uuid: "85e0fa583d4e4a7b86764d5ab97d067b"
format_stamp: "Formatted at 2026-06-19 09:01:15 on dist-test-slave-mtdr"
server_key: "1c0c6a09b8f4657b310fbb9d0177f1fe"
server_key_iv: "4cc45bf5571d226c9f907433f2273f42"
server_key_version: "encryptionkey@0"
I20260619 09:01:15.394452 2813 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260619 09:01:15.395192 2828 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 09:01:15.395387 2813 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260619 09:01:15.395486 2813 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781859658134510-392-0/minicluster-data/master-0/data,/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781859658134510-392-0/minicluster-data/master-0/wal
uuid: "85e0fa583d4e4a7b86764d5ab97d067b"
format_stamp: "Formatted at 2026-06-19 09:01:15 on dist-test-slave-mtdr"
server_key: "1c0c6a09b8f4657b310fbb9d0177f1fe"
server_key_iv: "4cc45bf5571d226c9f907433f2273f42"
server_key_version: "encryptionkey@0"
I20260619 09:01:15.395551 2813 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781859658134510-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781859658134510-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781859658134510-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 09:01:15.411188 2813 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 09:01:15.412949 2813 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 09:01:15.413079 2813 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 09:01:15.416667 2813 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33817
I20260619 09:01:15.416718 2880 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33817 every 8 connection(s)
I20260619 09:01:15.417158 2813 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781859658134510-392-0/minicluster-data/master-0/data/info.pb
I20260619 09:01:15.417819 2881 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 09:01:15.420228 2881 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85e0fa583d4e4a7b86764d5ab97d067b: Bootstrap starting.
I20260619 09:01:15.420783 2881 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 85e0fa583d4e4a7b86764d5ab97d067b: Neither blocks nor log segments found. Creating new log.
I20260619 09:01:15.421051 2881 log.cc:824] T 00000000000000000000000000000000 P 85e0fa583d4e4a7b86764d5ab97d067b: Log is configured to *not* fsync() on all Append() calls
I20260619 09:01:15.421691 2881 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85e0fa583d4e4a7b86764d5ab97d067b: No bootstrap required, opened a new log
I20260619 09:01:15.422098 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ez5NB/build/release/bin/kudu as pid 2813
I20260619 09:01:15.422188 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781859658134510-392-0/minicluster-data/master-0/wal/instance
I20260619 09:01:15.422358 392 external_mini_cluster.cc:1428] Setting key 3626402392de4f511b2591b72b5ddbd4
I20260619 09:01:15.424204 2881 raft_consensus.cc:348] T 00000000000000000000000000000000 P 85e0fa583d4e4a7b86764d5ab97d067b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85e0fa583d4e4a7b86764d5ab97d067b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33817 } }
I20260619 09:01:15.424345 2881 raft_consensus.cc:374] T 00000000000000000000000000000000 P 85e0fa583d4e4a7b86764d5ab97d067b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 09:01:15.424379 2881 raft_consensus.cc:729] T 00000000000000000000000000000000 P 85e0fa583d4e4a7b86764d5ab97d067b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85e0fa583d4e4a7b86764d5ab97d067b, State: Initialized, Role: FOLLOWER
I20260619 09:01:15.424533 2881 consensus_queue.cc:260] T 00000000000000000000000000000000 P 85e0fa583d4e4a7b86764d5ab97d067b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85e0fa583d4e4a7b86764d5ab97d067b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33817 } }
I20260619 09:01:15.424623 2881 raft_consensus.cc:388] T 00000000000000000000000000000000 P 85e0fa583d4e4a7b86764d5ab97d067b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 09:01:15.424665 2881 raft_consensus.cc:482] T 00000000000000000000000000000000 P 85e0fa583d4e4a7b86764d5ab97d067b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 09:01:15.424715 2881 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 85e0fa583d4e4a7b86764d5ab97d067b [term 0 FOLLOWER]: Advancing to term 1
I20260619 09:01:15.425374 2881 raft_consensus.cc:504] T 00000000000000000000000000000000 P 85e0fa583d4e4a7b86764d5ab97d067b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85e0fa583d4e4a7b86764d5ab97d067b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33817 } }
I20260619 09:01:15.425477 2881 leader_election.cc:302] T 00000000000000000000000000000000 P 85e0fa583d4e4a7b86764d5ab97d067b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 85e0fa583d4e4a7b86764d5ab97d067b; no voters:
I20260619 09:01:15.425624 2881 leader_election.cc:288] T 00000000000000000000000000000000 P 85e0fa583d4e4a7b86764d5ab97d067b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 09:01:15.425663 2886 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 85e0fa583d4e4a7b86764d5ab97d067b [term 1 FOLLOWER]: Leader election won for term 1
I20260619 09:01:15.425817 2886 raft_consensus.cc:686] T 00000000000000000000000000000000 P 85e0fa583d4e4a7b86764d5ab97d067b [term 1 LEADER]: Becoming Leader. State: Replica: 85e0fa583d4e4a7b86764d5ab97d067b, State: Running, Role: LEADER
I20260619 09:01:15.425900 2881 sys_catalog.cc:564] T 00000000000000000000000000000000 P 85e0fa583d4e4a7b86764d5ab97d067b [sys.catalog]: configured and running, proceeding with master startup.
I20260619 09:01:15.425961 2886 consensus_queue.cc:237] T 00000000000000000000000000000000 P 85e0fa583d4e4a7b86764d5ab97d067b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85e0fa583d4e4a7b86764d5ab97d067b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33817 } }
I20260619 09:01:15.426299 2886 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85e0fa583d4e4a7b86764d5ab97d067b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 85e0fa583d4e4a7b86764d5ab97d067b. Latest consensus state: current_term: 1 leader_uuid: "85e0fa583d4e4a7b86764d5ab97d067b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85e0fa583d4e4a7b86764d5ab97d067b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33817 } } }
I20260619 09:01:15.426306 2888 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85e0fa583d4e4a7b86764d5ab97d067b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "85e0fa583d4e4a7b86764d5ab97d067b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85e0fa583d4e4a7b86764d5ab97d067b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33817 } } }
I20260619 09:01:15.426481 2886 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85e0fa583d4e4a7b86764d5ab97d067b [sys.catalog]: This master's current role is: LEADER
I20260619 09:01:15.426565 2888 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85e0fa583d4e4a7b86764d5ab97d067b [sys.catalog]: This master's current role is: LEADER
I20260619 09:01:15.426927 2895 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 09:01:15.427474 2895 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 09:01:15.429628 2895 catalog_manager.cc:1347] Generated new cluster ID: d19bd17ba9704385ab6cf95694a29983
I20260619 09:01:15.429684 2895 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 19 09:01:15 dist-test-slave-mtdr krb5kdc[2800](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859675, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 09:01:15.434186 2885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:15.422863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43688 (local address 127.0.98.62:33817)
0619 09:01:15.423041 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:15.423048 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:15.423075 (+ 27us) server_negotiation.cc:407] Connection header received
0619 09:01:15.424203 (+ 1128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:15.424211 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:15.424262 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:15.424381 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:01:15.425233 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:15.426376 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:15.427441 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:15.428077 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:15.428189 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:15.430692 (+ 2503us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:01:15.430695 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:01:15.430696 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:01:15.430707 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:01:15.432604 (+ 1897us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:01:15.432983 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:01:15.432987 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:01:15.432987 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:01:15.433114 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:01:15.433306 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:01:15.433307 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:01:15.433308 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:01:15.433593 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:01:15.433751 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0619 09:01:15.433830 (+ 79us) server_negotiation.cc:299] Negotiation successful
0619 09:01:15.433964 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":71,"threads_started":1}
I20260619 09:01:15.445386 2895 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 09:01:15.445875 2895 catalog_manager.cc:1504] Loading token signing keys...
I20260619 09:01:15.450402 2895 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 85e0fa583d4e4a7b86764d5ab97d067b: Generated new TSK 0
I20260619 09:01:15.450549 2895 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781859658134510-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781859658134510-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 09:01:15.477149 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ez5NB/build/release/bin/kudu
/tmp/dist-test-task4ez5NB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781859658134510-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781859658134510-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781859658134510-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781859658134510-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33817
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781859658134510-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781859658134510-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781859658134510-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781859658134510-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 09:01:15.554704 2907 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 09:01:15.554942 2907 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 09:01:15.554979 2907 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 09:01:15.556133 2907 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 09:01:15.556174 2907 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 09:01:15.556209 2907 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 09:01:15.558081 2907 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781859658134510-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781859658134510-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781859658134510-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781859658134510-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33817
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2907
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781859658134510-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 09:01:15.558320 2907 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 09:01:15.558514 2907 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 09:01:15.563221 2912 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:01:15.563232 2913 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:01:15.563441 2907 server_base.cc:1034] running on GCE node
W20260619 09:01:15.563490 2915 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:01:15.563665 2907 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 09:01:15.563880 2907 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 09:01:15.565026 2907 hybrid_clock.cc:648] HybridClock initialized: now 1781859675565006 us; error 32 us; skew 500 ppm
Jun 19 09:01:15 dist-test-slave-mtdr krb5kdc[2800](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859675, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 09:01:15.567531 2907 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 09:01:15.567911 2907 webserver.cc:460] Webserver started at http://127.0.98.1:43485/ using document root <none> and password file <none>
I20260619 09:01:15.568079 2907 fs_manager.cc:359] Metadata directory not provided
I20260619 09:01:15.568117 2907 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 09:01:15.568213 2907 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 09:01:15.569156 2907 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781859658134510-392-0/minicluster-data/ts-0/data/instance:
uuid: "a0d2fb1cc4c748f296e3586654ff47eb"
format_stamp: "Formatted at 2026-06-19 09:01:15 on dist-test-slave-mtdr"
server_key: "aa767a992cc693b7fd3c9192ed76457a"
server_key_iv: "17368ab224d9b81d24f5831fc2793fab"
server_key_version: "encryptionkey@0"
I20260619 09:01:15.569442 2907 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781859658134510-392-0/minicluster-data/ts-0/wal/instance:
uuid: "a0d2fb1cc4c748f296e3586654ff47eb"
format_stamp: "Formatted at 2026-06-19 09:01:15 on dist-test-slave-mtdr"
server_key: "aa767a992cc693b7fd3c9192ed76457a"
server_key_iv: "17368ab224d9b81d24f5831fc2793fab"
server_key_version: "encryptionkey@0"
I20260619 09:01:15.570565 2907 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 09:01:15.571290 2922 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 09:01:15.571432 2907 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260619 09:01:15.571512 2907 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781859658134510-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781859658134510-392-0/minicluster-data/ts-0/wal
uuid: "a0d2fb1cc4c748f296e3586654ff47eb"
format_stamp: "Formatted at 2026-06-19 09:01:15 on dist-test-slave-mtdr"
server_key: "aa767a992cc693b7fd3c9192ed76457a"
server_key_iv: "17368ab224d9b81d24f5831fc2793fab"
server_key_version: "encryptionkey@0"
I20260619 09:01:15.571568 2907 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781859658134510-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781859658134510-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781859658134510-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 09:01:15.593135 2907 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 09:01:15.594969 2907 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 09:01:15.595073 2907 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 09:01:15.595250 2907 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 09:01:15.595520 2907 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 09:01:15.595551 2907 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 09:01:15.595570 2907 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 09:01:15.595594 2907 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 09:01:15.600917 2907 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35625
I20260619 09:01:15.600986 3035 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35625 every 8 connection(s)
I20260619 09:01:15.601248 2907 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781859658134510-392-0/minicluster-data/ts-0/data/info.pb
I20260619 09:01:15.601608 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ez5NB/build/release/bin/kudu as pid 2907
I20260619 09:01:15.601678 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781859658134510-392-0/minicluster-data/ts-0/wal/instance
I20260619 09:01:15.601848 392 external_mini_cluster.cc:1428] Setting key 805c50b306ecb99dd716bbb8c75c6f50
Jun 19 09:01:15 dist-test-slave-mtdr krb5kdc[2800](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859675, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260619 09:01:15.610700 2885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:15.601997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56201 (local address 127.0.98.62:33817)
0619 09:01:15.602052 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:15.602055 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:15.602271 (+ 216us) server_negotiation.cc:407] Connection header received
0619 09:01:15.603259 (+ 988us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:15.603260 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:15.603276 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:15.603341 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:01:15.603909 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:15.604436 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:15.605466 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:15.605781 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:15.605865 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:15.608084 (+ 2219us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:01:15.608085 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:01:15.608085 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:01:15.608093 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:01:15.609426 (+ 1333us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:01:15.609836 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:01:15.609836 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:01:15.609837 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:01:15.609886 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:01:15.610091 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:01:15.610092 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:01:15.610092 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:01:15.610206 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:01:15.610271 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0619 09:01:15.610564 (+ 293us) server_negotiation.cc:299] Negotiation successful
0619 09:01:15.610635 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260619 09:01:15.610913 3038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:15.601981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33817 (local address 127.0.98.1:56201)
0619 09:01:15.602146 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0619 09:01:15.602186 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 09:01:15.603153 (+ 967us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:01:15.603396 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:01:15.603399 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:01:15.603639 (+ 240us) client_negotiation.cc:189] Negotiated authn=SASL
0619 09:01:15.603820 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:15.603825 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:15.604528 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:15.604529 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:15.605379 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:15.605382 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:15.605936 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:15.605937 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:15.606055 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:15.606528 (+ 473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 09:01:15.606536 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 09:01:15.607989 (+ 1453us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 09:01:15.609495 (+ 1506us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:01:15.609501 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:01:15.609503 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:01:15.609765 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:01:15.609927 (+ 162us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:01:15.609928 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:01:15.609928 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:01:15.610016 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:01:15.610268 (+ 252us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 09:01:15.610270 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 09:01:15.610430 (+ 160us) client_negotiation.cc:769] Sending connection context
0619 09:01:15.610542 (+ 112us) client_negotiation.cc:240] Negotiation successful
0619 09:01:15.610661 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":90,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260619 09:01:15.611442 3036 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33817
I20260619 09:01:15.611531 3036 heartbeater.cc:464] Registering TS with master...
I20260619 09:01:15.611687 3036 heartbeater.cc:511] Master 127.0.98.62:33817 requested a full tablet report, sending...
I20260619 09:01:15.612138 2845 ts_manager.cc:194] Registered new tserver with Master: a0d2fb1cc4c748f296e3586654ff47eb (127.0.98.1:35625)
I20260619 09:01:15.612723 2845 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:56201
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781859658134510-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781859658134510-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 09:01:15.623826 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ez5NB/build/release/bin/kudu
/tmp/dist-test-task4ez5NB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781859658134510-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781859658134510-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781859658134510-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781859658134510-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33817
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781859658134510-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781859658134510-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781859658134510-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781859658134510-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 09:01:15.705405 3041 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 09:01:15.705606 3041 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 09:01:15.705626 3041 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 09:01:15.706750 3041 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 09:01:15.706789 3041 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 09:01:15.706838 3041 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 09:01:15.708787 3041 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781859658134510-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781859658134510-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781859658134510-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781859658134510-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33817
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3041
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781859658134510-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 09:01:15.709021 3041 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 09:01:15.709285 3041 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 09:01:15.714387 3049 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:01:15.714481 3047 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:01:15.714636 3046 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:01:15.714643 3041 server_base.cc:1034] running on GCE node
I20260619 09:01:15.714915 3041 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 09:01:15.715112 3041 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 09:01:15.716287 3041 hybrid_clock.cc:648] HybridClock initialized: now 1781859675716273 us; error 28 us; skew 500 ppm
Jun 19 09:01:15 dist-test-slave-mtdr krb5kdc[2800](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859675, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 09:01:15.718662 3041 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 09:01:15.719113 3041 webserver.cc:460] Webserver started at http://127.0.98.2:34253/ using document root <none> and password file <none>
I20260619 09:01:15.719311 3041 fs_manager.cc:359] Metadata directory not provided
I20260619 09:01:15.719358 3041 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 09:01:15.719481 3041 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 09:01:15.720520 3041 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781859658134510-392-0/minicluster-data/ts-1/data/instance:
uuid: "4d70adaaa02a4f229d1f53ba2fcfa05d"
format_stamp: "Formatted at 2026-06-19 09:01:15 on dist-test-slave-mtdr"
server_key: "da63861a9b7fafdb43aa4f2a8d9b0ff7"
server_key_iv: "a047a5035eee2f160a5ba3383265b15d"
server_key_version: "encryptionkey@0"
I20260619 09:01:15.720853 3041 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781859658134510-392-0/minicluster-data/ts-1/wal/instance:
uuid: "4d70adaaa02a4f229d1f53ba2fcfa05d"
format_stamp: "Formatted at 2026-06-19 09:01:15 on dist-test-slave-mtdr"
server_key: "da63861a9b7fafdb43aa4f2a8d9b0ff7"
server_key_iv: "a047a5035eee2f160a5ba3383265b15d"
server_key_version: "encryptionkey@0"
I20260619 09:01:15.722069 3041 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 09:01:15.722671 3056 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 09:01:15.722867 3041 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 09:01:15.722954 3041 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781859658134510-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781859658134510-392-0/minicluster-data/ts-1/wal
uuid: "4d70adaaa02a4f229d1f53ba2fcfa05d"
format_stamp: "Formatted at 2026-06-19 09:01:15 on dist-test-slave-mtdr"
server_key: "da63861a9b7fafdb43aa4f2a8d9b0ff7"
server_key_iv: "a047a5035eee2f160a5ba3383265b15d"
server_key_version: "encryptionkey@0"
I20260619 09:01:15.723018 3041 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781859658134510-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781859658134510-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781859658134510-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 09:01:15.734328 3041 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 09:01:15.736334 3041 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 09:01:15.736465 3041 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 09:01:15.736654 3041 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 09:01:15.736930 3041 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 09:01:15.736963 3041 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 09:01:15.736989 3041 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 09:01:15.737005 3041 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 09:01:15.742352 3041 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40527
I20260619 09:01:15.742408 3169 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40527 every 8 connection(s)
I20260619 09:01:15.742674 3041 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781859658134510-392-0/minicluster-data/ts-1/data/info.pb
Jun 19 09:01:15 dist-test-slave-mtdr krb5kdc[2800](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859675, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 09:01:15.748548 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ez5NB/build/release/bin/kudu as pid 3041
I20260619 09:01:15.748626 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781859658134510-392-0/minicluster-data/ts-1/wal/instance
I20260619 09:01:15.748767 392 external_mini_cluster.cc:1428] Setting key f049ac30b15585f169806500a7b125dd
I20260619 09:01:15.751693 2885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:15.743406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48643 (local address 127.0.98.62:33817)
0619 09:01:15.743458 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:15.743460 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:15.743655 (+ 195us) server_negotiation.cc:407] Connection header received
0619 09:01:15.744602 (+ 947us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:15.744603 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:15.744615 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:15.744671 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:01:15.745291 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:15.745771 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:15.746725 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:15.746941 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:15.747009 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:15.749049 (+ 2040us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:01:15.749050 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:01:15.749051 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:01:15.749059 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:01:15.750301 (+ 1242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:01:15.750730 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:01:15.750732 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:01:15.750732 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:01:15.750786 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:01:15.751070 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:01:15.751070 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:01:15.751071 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:01:15.751172 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:01:15.751221 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0619 09:01:15.751549 (+ 328us) server_negotiation.cc:299] Negotiation successful
0619 09:01:15.751627 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260619 09:01:15.751868 3172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:15.743336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33817 (local address 127.0.98.2:48643)
0619 09:01:15.743523 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0619 09:01:15.743569 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 09:01:15.744496 (+ 927us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:01:15.744750 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:01:15.744754 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:01:15.745004 (+ 250us) client_negotiation.cc:189] Negotiated authn=SASL
0619 09:01:15.745197 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:15.745201 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:15.745848 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:15.745849 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:15.746644 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:15.746646 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:15.747009 (+ 363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:15.747009 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:15.747128 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:15.747572 (+ 444us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 09:01:15.747583 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 09:01:15.748937 (+ 1354us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 09:01:15.750374 (+ 1437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:01:15.750380 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:01:15.750383 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:01:15.750653 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:01:15.750873 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:01:15.750874 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:01:15.750874 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:01:15.751013 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:01:15.751222 (+ 209us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 09:01:15.751226 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 09:01:15.751379 (+ 153us) client_negotiation.cc:769] Sending connection context
0619 09:01:15.751494 (+ 115us) client_negotiation.cc:240] Negotiation successful
0619 09:01:15.751651 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":82,"threads_started":1}
I20260619 09:01:15.752254 3170 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33817
I20260619 09:01:15.752342 3170 heartbeater.cc:464] Registering TS with master...
I20260619 09:01:15.752517 3170 heartbeater.cc:511] Master 127.0.98.62:33817 requested a full tablet report, sending...
I20260619 09:01:15.752883 2845 ts_manager.cc:194] Registered new tserver with Master: 4d70adaaa02a4f229d1f53ba2fcfa05d (127.0.98.2:40527)
I20260619 09:01:15.753244 2845 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:48643
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781859658134510-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781859658134510-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 09:01:15.771456 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ez5NB/build/release/bin/kudu
/tmp/dist-test-task4ez5NB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781859658134510-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781859658134510-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781859658134510-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781859658134510-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33817
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781859658134510-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781859658134510-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781859658134510-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781859658134510-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 09:01:15.849841 3175 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 09:01:15.850035 3175 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 09:01:15.850057 3175 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 09:01:15.851157 3175 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 09:01:15.851192 3175 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 09:01:15.851222 3175 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 09:01:15.852610 3175 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781859658134510-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781859658134510-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781859658134510-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781859658134510-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33817
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3175
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781859658134510-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 09:01:15.852777 3175 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 09:01:15.852941 3175 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 09:01:15.857501 3183 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:01:15.857517 3180 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:01:15.857578 3181 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:01:15.857708 3175 server_base.cc:1034] running on GCE node
I20260619 09:01:15.857915 3175 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 09:01:15.858135 3175 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 09:01:15.859266 3175 hybrid_clock.cc:648] HybridClock initialized: now 1781859675859255 us; error 28 us; skew 500 ppm
Jun 19 09:01:15 dist-test-slave-mtdr krb5kdc[2800](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859675, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 09:01:15.861577 3175 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 09:01:15.861980 3175 webserver.cc:460] Webserver started at http://127.0.98.3:35347/ using document root <none> and password file <none>
I20260619 09:01:15.862152 3175 fs_manager.cc:359] Metadata directory not provided
I20260619 09:01:15.862190 3175 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 09:01:15.862278 3175 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 09:01:15.863250 3175 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781859658134510-392-0/minicluster-data/ts-2/data/instance:
uuid: "70bc77c614fb400d8bb23b207f2389a0"
format_stamp: "Formatted at 2026-06-19 09:01:15 on dist-test-slave-mtdr"
server_key: "1979bcc471cd34ea1e6257d0830f4eaf"
server_key_iv: "3ce783048fec5e410fcd1b6c14f70933"
server_key_version: "encryptionkey@0"
I20260619 09:01:15.863523 3175 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781859658134510-392-0/minicluster-data/ts-2/wal/instance:
uuid: "70bc77c614fb400d8bb23b207f2389a0"
format_stamp: "Formatted at 2026-06-19 09:01:15 on dist-test-slave-mtdr"
server_key: "1979bcc471cd34ea1e6257d0830f4eaf"
server_key_iv: "3ce783048fec5e410fcd1b6c14f70933"
server_key_version: "encryptionkey@0"
I20260619 09:01:15.864651 3175 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 09:01:15.865299 3190 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 09:01:15.865491 3175 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 09:01:15.865569 3175 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781859658134510-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781859658134510-392-0/minicluster-data/ts-2/wal
uuid: "70bc77c614fb400d8bb23b207f2389a0"
format_stamp: "Formatted at 2026-06-19 09:01:15 on dist-test-slave-mtdr"
server_key: "1979bcc471cd34ea1e6257d0830f4eaf"
server_key_iv: "3ce783048fec5e410fcd1b6c14f70933"
server_key_version: "encryptionkey@0"
I20260619 09:01:15.865619 3175 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781859658134510-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781859658134510-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781859658134510-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 09:01:15.873526 3175 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 09:01:15.875488 3175 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 09:01:15.875586 3175 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 09:01:15.875742 3175 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 09:01:15.875968 3175 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 09:01:15.875995 3175 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 09:01:15.876013 3175 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 09:01:15.876030 3175 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 09:01:15.881057 3175 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:46083
I20260619 09:01:15.881129 3303 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:46083 every 8 connection(s)
I20260619 09:01:15.881372 3175 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781859658134510-392-0/minicluster-data/ts-2/data/info.pb
I20260619 09:01:15.885504 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ez5NB/build/release/bin/kudu as pid 3175
I20260619 09:01:15.885586 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781859658134510-392-0/minicluster-data/ts-2/wal/instance
I20260619 09:01:15.885739 392 external_mini_cluster.cc:1428] Setting key 335396ee5be71ec034487dfaa9256485
Jun 19 09:01:15 dist-test-slave-mtdr krb5kdc[2800](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859675, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 09:01:15.889801 2885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:15.881979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43043 (local address 127.0.98.62:33817)
0619 09:01:15.882021 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:15.882024 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:15.882192 (+ 168us) server_negotiation.cc:407] Connection header received
0619 09:01:15.883140 (+ 948us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:15.883142 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:15.883156 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:15.883207 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:01:15.883757 (+ 550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:15.884244 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:15.885185 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:15.885392 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:15.885444 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:15.887589 (+ 2145us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:01:15.887591 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:01:15.887591 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:01:15.887598 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:01:15.888661 (+ 1063us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:01:15.888989 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:01:15.888990 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:01:15.888990 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:01:15.889042 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:01:15.889244 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:01:15.889246 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:01:15.889246 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:01:15.889351 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:01:15.889404 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0619 09:01:15.889642 (+ 238us) server_negotiation.cc:299] Negotiation successful
0619 09:01:15.889715 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260619 09:01:15.889947 3306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:15.881940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33817 (local address 127.0.98.3:43043)
0619 09:01:15.882082 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0619 09:01:15.882117 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0619 09:01:15.883036 (+ 919us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:01:15.883270 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:01:15.883274 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:01:15.883500 (+ 226us) client_negotiation.cc:189] Negotiated authn=SASL
0619 09:01:15.883670 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:15.883674 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:15.884319 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:15.884320 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:15.885093 (+ 773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:15.885095 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:15.885454 (+ 359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:15.885455 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:15.885577 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:15.886086 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 09:01:15.886095 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0619 09:01:15.887504 (+ 1409us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 09:01:15.888729 (+ 1225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:01:15.888734 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:01:15.888737 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:01:15.888928 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:01:15.889094 (+ 166us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:01:15.889095 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:01:15.889095 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:01:15.889192 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:01:15.889402 (+ 210us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 09:01:15.889407 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 09:01:15.889531 (+ 124us) client_negotiation.cc:769] Sending connection context
0619 09:01:15.889614 (+ 83us) client_negotiation.cc:240] Negotiation successful
0619 09:01:15.889750 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":64,"threads_started":1}
I20260619 09:01:15.890306 3304 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33817
I20260619 09:01:15.890388 3304 heartbeater.cc:464] Registering TS with master...
I20260619 09:01:15.890508 3304 heartbeater.cc:511] Master 127.0.98.62:33817 requested a full tablet report, sending...
I20260619 09:01:15.890839 2845 ts_manager.cc:194] Registered new tserver with Master: 70bc77c614fb400d8bb23b207f2389a0 (127.0.98.3:46083)
I20260619 09:01:15.891167 2845 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:43043
I20260619 09:01:15.897166 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260619 09:01:15.907441 2885 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 09:01:15.906522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43696 (local address 127.0.98.62:33817)
0619 09:01:15.906577 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:15.906579 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:15.906677 (+ 98us) server_negotiation.cc:407] Connection header received
0619 09:01:15.906720 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:15.906720 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:15.906729 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:15.906785 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:01:15.907373 (+ 588us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43696: BlockingRecv error: recv got EOF from 127.0.0.1:43696 (error 108)
Metrics: {"server-negotiator.queue_time_us":28}
I20260619 09:01:15.908413 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ez5NB/build/release/bin/kudu with pid 2907
I20260619 09:01:15.912242 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ez5NB/build/release/bin/kudu with pid 3041
I20260619 09:01:15.915860 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ez5NB/build/release/bin/kudu with pid 3175
I20260619 09:01:15.919269 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ez5NB/build/release/bin/kudu with pid 2813
2026-06-19T09:01:15Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (1676 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781859658134510-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 09:01:15 dist-test-slave-mtdr krb5kdc[3316](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 09:01:15 dist-test-slave-mtdr krb5kdc[3316](info): set up 2 sockets
Jun 19 09:01:15 dist-test-slave-mtdr krb5kdc[3316](info): commencing operation
krb5kdc: starting...
W20260619 09:01:16.874265 392 mini_kdc.cc:121] Time spent starting KDC: real 0.937s user 0.001s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781859658134510-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781859658134510-392-0/krb5kdc/test-user.keytab.
Jun 19 09:01:16 dist-test-slave-mtdr krb5kdc[3316](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859676, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T09:01:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T09:01:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781859658134510-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781859658134510-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 09:01:16.984840 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ez5NB/build/release/bin/kudu
/tmp/dist-test-task4ez5NB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781859658134510-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781859658134510-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781859658134510-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781859658134510-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43555
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781859658134510-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43555
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781859658134510-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781859658134510-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781859658134510-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 09:01:17.061599 3329 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 09:01:17.061813 3329 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 09:01:17.061843 3329 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 09:01:17.062999 3329 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 09:01:17.063038 3329 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 09:01:17.063056 3329 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 09:01:17.063072 3329 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 09:01:17.063087 3329 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 09:01:17.064597 3329 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781859658134510-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781859658134510-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43555
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781859658134510-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43555
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781859658134510-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3329
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781859658134510-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 09:01:17.064836 3329 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 09:01:17.065033 3329 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 09:01:17.069425 3337 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:01:17.069437 3335 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:01:17.069480 3334 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:01:17.069792 3329 server_base.cc:1034] running on GCE node
I20260619 09:01:17.070011 3329 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 09:01:17.070266 3329 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 09:01:17.071398 3329 hybrid_clock.cc:648] HybridClock initialized: now 1781859677071383 us; error 28 us; skew 500 ppm
Jun 19 09:01:17 dist-test-slave-mtdr krb5kdc[3316](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859677, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 09:01:17.073892 3329 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 09:01:17.074304 3329 webserver.cc:460] Webserver started at http://127.0.98.62:33891/ using document root <none> and password file <none>
I20260619 09:01:17.074497 3329 fs_manager.cc:359] Metadata directory not provided
I20260619 09:01:17.074544 3329 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 09:01:17.074666 3329 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 09:01:17.075743 3329 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781859658134510-392-0/minicluster-data/master-0/data/instance:
uuid: "d6769942933749cc8299206aaa244aba"
format_stamp: "Formatted at 2026-06-19 09:01:17 on dist-test-slave-mtdr"
server_key: "bedb443046c7518361cfd2abdf391f67"
server_key_iv: "0dbbc8f110e9620fdf373f83bfb4880e"
server_key_version: "encryptionkey@0"
I20260619 09:01:17.076077 3329 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781859658134510-392-0/minicluster-data/master-0/wal/instance:
uuid: "d6769942933749cc8299206aaa244aba"
format_stamp: "Formatted at 2026-06-19 09:01:17 on dist-test-slave-mtdr"
server_key: "bedb443046c7518361cfd2abdf391f67"
server_key_iv: "0dbbc8f110e9620fdf373f83bfb4880e"
server_key_version: "encryptionkey@0"
I20260619 09:01:17.077294 3329 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 09:01:17.077909 3344 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 09:01:17.078068 3329 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 09:01:17.078148 3329 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781859658134510-392-0/minicluster-data/master-0/data,/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781859658134510-392-0/minicluster-data/master-0/wal
uuid: "d6769942933749cc8299206aaa244aba"
format_stamp: "Formatted at 2026-06-19 09:01:17 on dist-test-slave-mtdr"
server_key: "bedb443046c7518361cfd2abdf391f67"
server_key_iv: "0dbbc8f110e9620fdf373f83bfb4880e"
server_key_version: "encryptionkey@0"
I20260619 09:01:17.078213 3329 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781859658134510-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781859658134510-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781859658134510-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 09:01:17.110520 3329 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 09:01:17.112552 3329 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 09:01:17.112684 3329 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 09:01:17.116257 3329 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43555
I20260619 09:01:17.116299 3396 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43555 every 8 connection(s)
I20260619 09:01:17.116613 3329 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781859658134510-392-0/minicluster-data/master-0/data/info.pb
I20260619 09:01:17.117460 3397 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 09:01:17.118990 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ez5NB/build/release/bin/kudu as pid 3329
I20260619 09:01:17.119076 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781859658134510-392-0/minicluster-data/master-0/wal/instance
I20260619 09:01:17.119256 392 external_mini_cluster.cc:1428] Setting key 94f16e1a6ced7ba94be5f881f513354d
I20260619 09:01:17.119868 3397 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d6769942933749cc8299206aaa244aba: Bootstrap starting.
I20260619 09:01:17.120543 3397 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d6769942933749cc8299206aaa244aba: Neither blocks nor log segments found. Creating new log.
I20260619 09:01:17.120860 3397 log.cc:824] T 00000000000000000000000000000000 P d6769942933749cc8299206aaa244aba: Log is configured to *not* fsync() on all Append() calls
I20260619 09:01:17.121501 3397 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d6769942933749cc8299206aaa244aba: No bootstrap required, opened a new log
I20260619 09:01:17.123703 3397 raft_consensus.cc:348] T 00000000000000000000000000000000 P d6769942933749cc8299206aaa244aba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6769942933749cc8299206aaa244aba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43555 } }
I20260619 09:01:17.123870 3397 raft_consensus.cc:374] T 00000000000000000000000000000000 P d6769942933749cc8299206aaa244aba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 09:01:17.123904 3397 raft_consensus.cc:729] T 00000000000000000000000000000000 P d6769942933749cc8299206aaa244aba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6769942933749cc8299206aaa244aba, State: Initialized, Role: FOLLOWER
I20260619 09:01:17.124025 3397 consensus_queue.cc:260] T 00000000000000000000000000000000 P d6769942933749cc8299206aaa244aba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6769942933749cc8299206aaa244aba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43555 } }
I20260619 09:01:17.124104 3397 raft_consensus.cc:388] T 00000000000000000000000000000000 P d6769942933749cc8299206aaa244aba [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 09:01:17.124131 3397 raft_consensus.cc:482] T 00000000000000000000000000000000 P d6769942933749cc8299206aaa244aba [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 09:01:17.124176 3397 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d6769942933749cc8299206aaa244aba [term 0 FOLLOWER]: Advancing to term 1
I20260619 09:01:17.124792 3397 raft_consensus.cc:504] T 00000000000000000000000000000000 P d6769942933749cc8299206aaa244aba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6769942933749cc8299206aaa244aba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43555 } }
I20260619 09:01:17.124890 3397 leader_election.cc:302] T 00000000000000000000000000000000 P d6769942933749cc8299206aaa244aba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d6769942933749cc8299206aaa244aba; no voters:
I20260619 09:01:17.125036 3397 leader_election.cc:288] T 00000000000000000000000000000000 P d6769942933749cc8299206aaa244aba [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 09:01:17.125072 3402 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d6769942933749cc8299206aaa244aba [term 1 FOLLOWER]: Leader election won for term 1
I20260619 09:01:17.125178 3402 raft_consensus.cc:686] T 00000000000000000000000000000000 P d6769942933749cc8299206aaa244aba [term 1 LEADER]: Becoming Leader. State: Replica: d6769942933749cc8299206aaa244aba, State: Running, Role: LEADER
I20260619 09:01:17.125288 3397 sys_catalog.cc:564] T 00000000000000000000000000000000 P d6769942933749cc8299206aaa244aba [sys.catalog]: configured and running, proceeding with master startup.
I20260619 09:01:17.125319 3402 consensus_queue.cc:237] T 00000000000000000000000000000000 P d6769942933749cc8299206aaa244aba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6769942933749cc8299206aaa244aba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43555 } }
I20260619 09:01:17.125653 3402 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6769942933749cc8299206aaa244aba [sys.catalog]: SysCatalogTable state changed. Reason: New leader d6769942933749cc8299206aaa244aba. Latest consensus state: current_term: 1 leader_uuid: "d6769942933749cc8299206aaa244aba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6769942933749cc8299206aaa244aba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43555 } } }
I20260619 09:01:17.125739 3402 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6769942933749cc8299206aaa244aba [sys.catalog]: This master's current role is: LEADER
I20260619 09:01:17.125990 3404 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6769942933749cc8299206aaa244aba [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d6769942933749cc8299206aaa244aba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6769942933749cc8299206aaa244aba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43555 } } }
I20260619 09:01:17.126289 3404 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6769942933749cc8299206aaa244aba [sys.catalog]: This master's current role is: LEADER
Jun 19 09:01:17 dist-test-slave-mtdr krb5kdc[3316](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859676, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 09:01:17.126499 3409 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 09:01:17.127082 3409 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 09:01:17.128856 3409 catalog_manager.cc:1347] Generated new cluster ID: c8a8ba3cd0814d5a80e64b7e853889e8
I20260619 09:01:17.128921 3409 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 09:01:17.129909 3400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:17.119708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49640 (local address 127.0.98.62:43555)
0619 09:01:17.119905 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:17.119914 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:17.119946 (+ 32us) server_negotiation.cc:407] Connection header received
0619 09:01:17.120470 (+ 524us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:17.120479 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:17.120543 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:17.120696 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:01:17.121283 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:17.122620 (+ 1337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:17.123349 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:17.123953 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:17.124060 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:17.126867 (+ 2807us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:01:17.126871 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:01:17.126871 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:01:17.126882 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:01:17.128506 (+ 1624us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:01:17.128856 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:01:17.128859 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:01:17.128859 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:01:17.128981 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:01:17.129150 (+ 169us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:01:17.129151 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:01:17.129151 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:01:17.129378 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:01:17.129516 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0619 09:01:17.129608 (+ 92us) server_negotiation.cc:299] Negotiation successful
0619 09:01:17.129731 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":82,"threads_started":1}
I20260619 09:01:17.135224 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 09:01:17.148104 3409 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 09:01:17.148710 3409 catalog_manager.cc:1504] Loading token signing keys...
I20260619 09:01:17.151234 3409 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d6769942933749cc8299206aaa244aba: Generated new TSK 0
I20260619 09:01:17.151410 3409 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 09:01:17.152158 3400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:17.147805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49644 (local address 127.0.98.62:43555)
0619 09:01:17.147850 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:17.147852 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:17.147970 (+ 118us) server_negotiation.cc:407] Connection header received
0619 09:01:17.148063 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:17.148063 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:17.148075 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:17.148122 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:01:17.148586 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:17.149318 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:17.150350 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:17.150678 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:17.150747 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:17.151729 (+ 982us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:01:17.151730 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:01:17.151730 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 09:01:17.151903 (+ 173us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:01:17.151921 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 09:01:17.151927 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:01:17.151980 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0619 09:01:17.152050 (+ 70us) server_negotiation.cc:299] Negotiation successful
0619 09:01:17.152086 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260619 09:01:17.154304 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ez5NB/build/release/bin/kudu with pid 3329
2026-06-19T09:01:17Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1233 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781859658134510-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 09:01:17 dist-test-slave-mtdr krb5kdc[3430](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 09:01:17 dist-test-slave-mtdr krb5kdc[3430](info): set up 2 sockets
Jun 19 09:01:17 dist-test-slave-mtdr krb5kdc[3430](info): commencing operation
krb5kdc: starting...
W20260619 09:01:18.106336 392 mini_kdc.cc:121] Time spent starting KDC: real 0.935s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781859658134510-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781859658134510-392-0/krb5kdc/test-user.keytab.
Jun 19 09:01:18 dist-test-slave-mtdr krb5kdc[3430](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859678, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T09:01:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T09:01:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781859658134510-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781859658134510-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 09:01:18.215492 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ez5NB/build/release/bin/kudu
/tmp/dist-test-task4ez5NB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781859658134510-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781859658134510-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781859658134510-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781859658134510-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43509
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781859658134510-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32891
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43509
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781859658134510-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781859658134510-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781859658134510-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 09:01:18.291921 3443 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 09:01:18.292114 3443 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 09:01:18.292143 3443 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 09:01:18.293231 3443 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 09:01:18.293267 3443 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 09:01:18.293284 3443 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 09:01:18.293299 3443 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 09:01:18.293316 3443 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 09:01:18.294729 3443 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32891
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781859658134510-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781859658134510-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43509
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781859658134510-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43509
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781859658134510-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3443
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781859658134510-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 09:01:18.294986 3443 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 09:01:18.295181 3443 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 09:01:18.299507 3451 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:01:18.299582 3449 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:01:18.299513 3448 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:01:18.299930 3443 server_base.cc:1034] running on GCE node
I20260619 09:01:18.300165 3443 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 09:01:18.300410 3443 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 09:01:18.301543 3443 hybrid_clock.cc:648] HybridClock initialized: now 1781859678301528 us; error 30 us; skew 500 ppm
Jun 19 09:01:18 dist-test-slave-mtdr krb5kdc[3430](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859678, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 09:01:18.303951 3443 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 09:01:18.304397 3443 webserver.cc:460] Webserver started at http://127.0.98.62:39141/ using document root <none> and password file <none>
I20260619 09:01:18.304592 3443 fs_manager.cc:359] Metadata directory not provided
I20260619 09:01:18.304638 3443 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 09:01:18.304731 3443 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 09:01:18.305733 3443 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781859658134510-392-0/minicluster-data/master-0/data/instance:
uuid: "6e49707cec1b421a9e73dd9867961770"
format_stamp: "Formatted at 2026-06-19 09:01:18 on dist-test-slave-mtdr"
server_key: "53983a7bb3ccc34783c97bd4caa54829"
server_key_iv: "da2e63249a52de760dcfac15cabd36ab"
server_key_version: "encryptionkey@0"
I20260619 09:01:18.306069 3443 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781859658134510-392-0/minicluster-data/master-0/wal/instance:
uuid: "6e49707cec1b421a9e73dd9867961770"
format_stamp: "Formatted at 2026-06-19 09:01:18 on dist-test-slave-mtdr"
server_key: "53983a7bb3ccc34783c97bd4caa54829"
server_key_iv: "da2e63249a52de760dcfac15cabd36ab"
server_key_version: "encryptionkey@0"
I20260619 09:01:18.307415 3443 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260619 09:01:18.308147 3458 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 09:01:18.308298 3443 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 09:01:18.308379 3443 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781859658134510-392-0/minicluster-data/master-0/data,/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781859658134510-392-0/minicluster-data/master-0/wal
uuid: "6e49707cec1b421a9e73dd9867961770"
format_stamp: "Formatted at 2026-06-19 09:01:18 on dist-test-slave-mtdr"
server_key: "53983a7bb3ccc34783c97bd4caa54829"
server_key_iv: "da2e63249a52de760dcfac15cabd36ab"
server_key_version: "encryptionkey@0"
I20260619 09:01:18.308449 3443 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781859658134510-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781859658134510-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781859658134510-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 09:01:18.325305 3443 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 09:01:18.327389 3443 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 09:01:18.327505 3443 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 09:01:18.331154 3443 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43509
I20260619 09:01:18.331197 3510 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43509 every 8 connection(s)
I20260619 09:01:18.331486 3443 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781859658134510-392-0/minicluster-data/master-0/data/info.pb
I20260619 09:01:18.332404 3511 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 09:01:18.334578 3511 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6e49707cec1b421a9e73dd9867961770: Bootstrap starting.
I20260619 09:01:18.335146 3511 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6e49707cec1b421a9e73dd9867961770: Neither blocks nor log segments found. Creating new log.
I20260619 09:01:18.335386 3511 log.cc:824] T 00000000000000000000000000000000 P 6e49707cec1b421a9e73dd9867961770: Log is configured to *not* fsync() on all Append() calls
I20260619 09:01:18.336059 3511 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6e49707cec1b421a9e73dd9867961770: No bootstrap required, opened a new log
I20260619 09:01:18.337545 3511 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6e49707cec1b421a9e73dd9867961770 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e49707cec1b421a9e73dd9867961770" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43509 } }
I20260619 09:01:18.337673 3511 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6e49707cec1b421a9e73dd9867961770 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 09:01:18.337694 3511 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6e49707cec1b421a9e73dd9867961770 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e49707cec1b421a9e73dd9867961770, State: Initialized, Role: FOLLOWER
I20260619 09:01:18.337824 3511 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6e49707cec1b421a9e73dd9867961770 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e49707cec1b421a9e73dd9867961770" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43509 } }
I20260619 09:01:18.337893 3511 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6e49707cec1b421a9e73dd9867961770 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 09:01:18.337924 3511 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6e49707cec1b421a9e73dd9867961770 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 09:01:18.337956 3511 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6e49707cec1b421a9e73dd9867961770 [term 0 FOLLOWER]: Advancing to term 1
I20260619 09:01:18.338685 3511 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6e49707cec1b421a9e73dd9867961770 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e49707cec1b421a9e73dd9867961770" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43509 } }
I20260619 09:01:18.338764 3511 leader_election.cc:302] T 00000000000000000000000000000000 P 6e49707cec1b421a9e73dd9867961770 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6e49707cec1b421a9e73dd9867961770; no voters:
I20260619 09:01:18.338925 3511 leader_election.cc:288] T 00000000000000000000000000000000 P 6e49707cec1b421a9e73dd9867961770 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 09:01:18.338971 3514 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6e49707cec1b421a9e73dd9867961770 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 09:01:18.339111 3514 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6e49707cec1b421a9e73dd9867961770 [term 1 LEADER]: Becoming Leader. State: Replica: 6e49707cec1b421a9e73dd9867961770, State: Running, Role: LEADER
I20260619 09:01:18.339159 3511 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6e49707cec1b421a9e73dd9867961770 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 09:01:18.339222 3514 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6e49707cec1b421a9e73dd9867961770 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e49707cec1b421a9e73dd9867961770" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43509 } }
I20260619 09:01:18.339564 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ez5NB/build/release/bin/kudu as pid 3443
I20260619 09:01:18.339581 3516 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e49707cec1b421a9e73dd9867961770 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6e49707cec1b421a9e73dd9867961770. Latest consensus state: current_term: 1 leader_uuid: "6e49707cec1b421a9e73dd9867961770" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e49707cec1b421a9e73dd9867961770" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43509 } } }
I20260619 09:01:18.339591 3515 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e49707cec1b421a9e73dd9867961770 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6e49707cec1b421a9e73dd9867961770" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e49707cec1b421a9e73dd9867961770" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43509 } } }
I20260619 09:01:18.339653 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781859658134510-392-0/minicluster-data/master-0/wal/instance
I20260619 09:01:18.339664 3516 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e49707cec1b421a9e73dd9867961770 [sys.catalog]: This master's current role is: LEADER
I20260619 09:01:18.339668 3515 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e49707cec1b421a9e73dd9867961770 [sys.catalog]: This master's current role is: LEADER
I20260619 09:01:18.339815 392 external_mini_cluster.cc:1428] Setting key 79b2105199e6e96da9e351fee08f6203
I20260619 09:01:18.339972 3521 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 09:01:18.340631 3521 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 09:01:18.342001 3521 catalog_manager.cc:1347] Generated new cluster ID: da2961a569cc41adabc233b9aadaf870
I20260619 09:01:18.342051 3521 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 19 09:01:18 dist-test-slave-mtdr krb5kdc[3430](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859678, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 09:01:18.350188 3531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:18.340385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44906 (local address 127.0.98.62:43509)
0619 09:01:18.340652 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:18.340661 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:18.340692 (+ 31us) server_negotiation.cc:407] Connection header received
0619 09:01:18.340800 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:18.340807 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:18.340860 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:18.341718 (+ 858us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:01:18.341805 (+ 87us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:18.343182 (+ 1377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:18.343944 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:18.344597 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:18.344685 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:18.346852 (+ 2167us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:01:18.346855 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:01:18.346856 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:01:18.346867 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:01:18.348670 (+ 1803us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:01:18.348981 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:01:18.348984 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:01:18.348984 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:01:18.349097 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:01:18.349263 (+ 166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:01:18.349264 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:01:18.349264 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:01:18.349568 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:01:18.349718 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0619 09:01:18.349828 (+ 110us) server_negotiation.cc:299] Negotiation successful
0619 09:01:18.349976 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":223,"thread_start_us":72,"threads_started":1}
I20260619 09:01:18.360867 3521 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 09:01:18.361338 3521 catalog_manager.cc:1504] Loading token signing keys...
I20260619 09:01:18.369086 3521 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6e49707cec1b421a9e73dd9867961770: Generated new TSK 0
I20260619 09:01:18.369251 3521 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781859658134510-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781859658134510-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 09:01:18.396982 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ez5NB/build/release/bin/kudu
/tmp/dist-test-task4ez5NB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781859658134510-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781859658134510-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781859658134510-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781859658134510-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43509
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781859658134510-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32891
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781859658134510-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781859658134510-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781859658134510-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 09:01:18.491519 3537 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 09:01:18.491701 3537 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 09:01:18.491724 3537 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 09:01:18.492791 3537 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 09:01:18.492821 3537 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 09:01:18.492846 3537 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 09:01:18.494174 3537 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32891
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781859658134510-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781859658134510-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781859658134510-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781859658134510-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43509
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3537
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781859658134510-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 09:01:18.494343 3537 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 09:01:18.494517 3537 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 09:01:18.499148 3545 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:01:18.499053 3542 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:01:18.499056 3543 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:01:18.499092 3537 server_base.cc:1034] running on GCE node
I20260619 09:01:18.499488 3537 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 09:01:18.499708 3537 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 09:01:18.500849 3537 hybrid_clock.cc:648] HybridClock initialized: now 1781859678500835 us; error 27 us; skew 500 ppm
Jun 19 09:01:18 dist-test-slave-mtdr krb5kdc[3430](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859678, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 09:01:18.503196 3537 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 09:01:18.503573 3537 webserver.cc:460] Webserver started at http://127.0.98.1:43261/ using document root <none> and password file <none>
I20260619 09:01:18.503758 3537 fs_manager.cc:359] Metadata directory not provided
I20260619 09:01:18.503803 3537 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 09:01:18.503916 3537 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 09:01:18.504889 3537 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781859658134510-392-0/minicluster-data/ts-0/data/instance:
uuid: "3ad91140a8b941c89722a4a0309ce73e"
format_stamp: "Formatted at 2026-06-19 09:01:18 on dist-test-slave-mtdr"
server_key: "fd8300969847a5b928f85accb0d665d8"
server_key_iv: "3afc0b06b44825f064cbb18ee24bb4ce"
server_key_version: "encryptionkey@0"
I20260619 09:01:18.505195 3537 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781859658134510-392-0/minicluster-data/ts-0/wal/instance:
uuid: "3ad91140a8b941c89722a4a0309ce73e"
format_stamp: "Formatted at 2026-06-19 09:01:18 on dist-test-slave-mtdr"
server_key: "fd8300969847a5b928f85accb0d665d8"
server_key_iv: "3afc0b06b44825f064cbb18ee24bb4ce"
server_key_version: "encryptionkey@0"
I20260619 09:01:18.506358 3537 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 09:01:18.507050 3552 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 09:01:18.507193 3537 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 09:01:18.507270 3537 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781859658134510-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781859658134510-392-0/minicluster-data/ts-0/wal
uuid: "3ad91140a8b941c89722a4a0309ce73e"
format_stamp: "Formatted at 2026-06-19 09:01:18 on dist-test-slave-mtdr"
server_key: "fd8300969847a5b928f85accb0d665d8"
server_key_iv: "3afc0b06b44825f064cbb18ee24bb4ce"
server_key_version: "encryptionkey@0"
I20260619 09:01:18.507324 3537 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781859658134510-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781859658134510-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781859658134510-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 09:01:18.517776 3537 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 09:01:18.519533 3537 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 09:01:18.519650 3537 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 09:01:18.519831 3537 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 09:01:18.520093 3537 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 09:01:18.520124 3537 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 09:01:18.520148 3537 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 09:01:18.520174 3537 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 09:01:18.525444 3537 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46859
I20260619 09:01:18.525482 3665 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46859 every 8 connection(s)
I20260619 09:01:18.525753 3537 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781859658134510-392-0/minicluster-data/ts-0/data/info.pb
Jun 19 09:01:18 dist-test-slave-mtdr krb5kdc[3430](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859678, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 09:01:18.531674 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ez5NB/build/release/bin/kudu as pid 3537
I20260619 09:01:18.531769 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781859658134510-392-0/minicluster-data/ts-0/wal/instance
I20260619 09:01:18.531935 392 external_mini_cluster.cc:1428] Setting key d7a92abcb26d8f9302d270e69afc4ff2
I20260619 09:01:18.534325 3531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:18.526363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56897 (local address 127.0.98.62:43509)
0619 09:01:18.526402 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:18.526404 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:18.526608 (+ 204us) server_negotiation.cc:407] Connection header received
0619 09:01:18.527168 (+ 560us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:18.527169 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:18.527181 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:18.527223 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:01:18.528069 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:18.528563 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:18.529503 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:18.529707 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:18.529755 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:18.531716 (+ 1961us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:01:18.531718 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:01:18.531718 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:01:18.531731 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:01:18.533136 (+ 1405us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:01:18.533472 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:01:18.533473 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:01:18.533473 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:01:18.533525 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:01:18.533739 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:01:18.533740 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:01:18.533740 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:01:18.533861 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:01:18.533918 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0619 09:01:18.534201 (+ 283us) server_negotiation.cc:299] Negotiation successful
0619 09:01:18.534263 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260619 09:01:18.534531 3668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:18.526338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43509 (local address 127.0.98.1:56897)
0619 09:01:18.526485 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0619 09:01:18.526524 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 09:01:18.527073 (+ 549us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:01:18.527289 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:01:18.527292 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:01:18.527809 (+ 517us) client_negotiation.cc:189] Negotiated authn=SASL
0619 09:01:18.527997 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:18.528001 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:18.528623 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:18.528624 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:18.529437 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:18.529439 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:18.529763 (+ 324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:18.529763 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:18.529851 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:18.530296 (+ 445us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 09:01:18.530304 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 09:01:18.531619 (+ 1315us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 09:01:18.533211 (+ 1592us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:01:18.533216 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:01:18.533218 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:01:18.533406 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:01:18.533585 (+ 179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:01:18.533586 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:01:18.533586 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:01:18.533675 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:01:18.533917 (+ 242us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 09:01:18.533922 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 09:01:18.534079 (+ 157us) client_negotiation.cc:769] Sending connection context
0619 09:01:18.534179 (+ 100us) client_negotiation.cc:240] Negotiation successful
0619 09:01:18.534328 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":69,"threads_started":1}
I20260619 09:01:18.535117 3666 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43509
I20260619 09:01:18.535200 3666 heartbeater.cc:464] Registering TS with master...
I20260619 09:01:18.535362 3666 heartbeater.cc:511] Master 127.0.98.62:43509 requested a full tablet report, sending...
I20260619 09:01:18.535851 3475 ts_manager.cc:194] Registered new tserver with Master: 3ad91140a8b941c89722a4a0309ce73e (127.0.98.1:46859)
I20260619 09:01:18.536547 3475 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:56897
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781859658134510-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781859658134510-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 09:01:18.555124 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ez5NB/build/release/bin/kudu
/tmp/dist-test-task4ez5NB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781859658134510-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781859658134510-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781859658134510-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781859658134510-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43509
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781859658134510-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32891
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781859658134510-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781859658134510-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781859658134510-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 09:01:18.633916 3671 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 09:01:18.634179 3671 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 09:01:18.634213 3671 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 09:01:18.635362 3671 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 09:01:18.635398 3671 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 09:01:18.635432 3671 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 09:01:18.636797 3671 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32891
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781859658134510-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781859658134510-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781859658134510-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781859658134510-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43509
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3671
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781859658134510-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 09:01:18.637017 3671 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 09:01:18.637218 3671 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 09:01:18.642069 3676 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:01:18.642058 3679 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:01:18.642060 3677 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:01:18.642329 3671 server_base.cc:1034] running on GCE node
I20260619 09:01:18.642518 3671 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 09:01:18.642750 3671 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 09:01:18.643904 3671 hybrid_clock.cc:648] HybridClock initialized: now 1781859678643880 us; error 36 us; skew 500 ppm
Jun 19 09:01:18 dist-test-slave-mtdr krb5kdc[3430](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859678, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 09:01:18.646301 3671 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 09:01:18.646719 3671 webserver.cc:460] Webserver started at http://127.0.98.2:44775/ using document root <none> and password file <none>
I20260619 09:01:18.646957 3671 fs_manager.cc:359] Metadata directory not provided
I20260619 09:01:18.647006 3671 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 09:01:18.647130 3671 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 09:01:18.648083 3671 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781859658134510-392-0/minicluster-data/ts-1/data/instance:
uuid: "9478df33dd0048b0a6d3e3bdbd001cf1"
format_stamp: "Formatted at 2026-06-19 09:01:18 on dist-test-slave-mtdr"
server_key: "aa69a8f3133f10bd9ad72cb9ec14bbbc"
server_key_iv: "5d538aaa85fedd9a516d8e5c8009b896"
server_key_version: "encryptionkey@0"
I20260619 09:01:18.648435 3671 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781859658134510-392-0/minicluster-data/ts-1/wal/instance:
uuid: "9478df33dd0048b0a6d3e3bdbd001cf1"
format_stamp: "Formatted at 2026-06-19 09:01:18 on dist-test-slave-mtdr"
server_key: "aa69a8f3133f10bd9ad72cb9ec14bbbc"
server_key_iv: "5d538aaa85fedd9a516d8e5c8009b896"
server_key_version: "encryptionkey@0"
I20260619 09:01:18.649592 3671 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 09:01:18.650192 3686 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 09:01:18.650370 3671 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 09:01:18.650447 3671 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781859658134510-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781859658134510-392-0/minicluster-data/ts-1/wal
uuid: "9478df33dd0048b0a6d3e3bdbd001cf1"
format_stamp: "Formatted at 2026-06-19 09:01:18 on dist-test-slave-mtdr"
server_key: "aa69a8f3133f10bd9ad72cb9ec14bbbc"
server_key_iv: "5d538aaa85fedd9a516d8e5c8009b896"
server_key_version: "encryptionkey@0"
I20260619 09:01:18.650511 3671 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781859658134510-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781859658134510-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781859658134510-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 09:01:18.668324 3671 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 09:01:18.670284 3671 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 09:01:18.670418 3671 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 09:01:18.670620 3671 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 09:01:18.670933 3671 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 09:01:18.670969 3671 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 09:01:18.670997 3671 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 09:01:18.671018 3671 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 09:01:18.676277 3671 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40099
I20260619 09:01:18.676318 3799 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40099 every 8 connection(s)
I20260619 09:01:18.676620 3671 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781859658134510-392-0/minicluster-data/ts-1/data/info.pb
I20260619 09:01:18.679529 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ez5NB/build/release/bin/kudu as pid 3671
I20260619 09:01:18.679615 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781859658134510-392-0/minicluster-data/ts-1/wal/instance
I20260619 09:01:18.679800 392 external_mini_cluster.cc:1428] Setting key 804382d939153a97b0fd0693c63e9196
Jun 19 09:01:18 dist-test-slave-mtdr krb5kdc[3430](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859678, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 09:01:18.685617 3531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:18.677280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51457 (local address 127.0.98.62:43509)
0619 09:01:18.677330 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:18.677332 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:18.677529 (+ 197us) server_negotiation.cc:407] Connection header received
0619 09:01:18.678027 (+ 498us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:18.678029 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:18.678040 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:18.678097 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:01:18.679027 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:18.679516 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:18.680850 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:18.681034 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:18.681087 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:18.683173 (+ 2086us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:01:18.683174 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:01:18.683175 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:01:18.683183 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:01:18.684413 (+ 1230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:01:18.684775 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:01:18.684777 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:01:18.684777 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:01:18.684826 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:01:18.685033 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:01:18.685034 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:01:18.685034 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:01:18.685138 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:01:18.685190 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0619 09:01:18.685444 (+ 254us) server_negotiation.cc:299] Negotiation successful
0619 09:01:18.685527 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260619 09:01:18.685793 3802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:18.677250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43509 (local address 127.0.98.2:51457)
0619 09:01:18.677409 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0619 09:01:18.677446 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0619 09:01:18.677918 (+ 472us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:01:18.678170 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:01:18.678173 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:01:18.678736 (+ 563us) client_negotiation.cc:189] Negotiated authn=SASL
0619 09:01:18.678949 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:18.678955 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:18.679613 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:18.679614 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:18.680771 (+ 1157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:18.680774 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:18.681108 (+ 334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:18.681109 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:18.681240 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:18.681746 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 09:01:18.681756 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 09:01:18.683091 (+ 1335us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 09:01:18.684486 (+ 1395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:01:18.684493 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:01:18.684495 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:01:18.684705 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:01:18.684878 (+ 173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:01:18.684878 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:01:18.684878 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:01:18.684980 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:01:18.685191 (+ 211us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 09:01:18.685197 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 09:01:18.685332 (+ 135us) client_negotiation.cc:769] Sending connection context
0619 09:01:18.685418 (+ 86us) client_negotiation.cc:240] Negotiation successful
0619 09:01:18.685572 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":79,"threads_started":1}
I20260619 09:01:18.686226 3800 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43509
I20260619 09:01:18.686313 3800 heartbeater.cc:464] Registering TS with master...
I20260619 09:01:18.686476 3800 heartbeater.cc:511] Master 127.0.98.62:43509 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260619 09:01:18.686866 3475 ts_manager.cc:194] Registered new tserver with Master: 9478df33dd0048b0a6d3e3bdbd001cf1 (127.0.98.2:40099)
I20260619 09:01:18.687268 3475 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:51457
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781859658134510-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781859658134510-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 09:01:18.703341 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ez5NB/build/release/bin/kudu
/tmp/dist-test-task4ez5NB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781859658134510-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781859658134510-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781859658134510-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781859658134510-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43509
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781859658134510-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32891
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781859658134510-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781859658134510-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781859658134510-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 09:01:18.780982 3805 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 09:01:18.781170 3805 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 09:01:18.781191 3805 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 09:01:18.782271 3805 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 09:01:18.782306 3805 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 09:01:18.782330 3805 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 09:01:18.783689 3805 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32891
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781859658134510-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781859658134510-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781859658134510-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781859658134510-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43509
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3805
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781859658134510-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 09:01:18.783864 3805 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 09:01:18.784024 3805 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 09:01:18.788697 3810 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:01:18.788704 3813 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:01:18.788763 3805 server_base.cc:1034] running on GCE node
W20260619 09:01:18.788669 3811 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:01:18.789058 3805 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 09:01:18.789265 3805 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 09:01:18.790391 3805 hybrid_clock.cc:648] HybridClock initialized: now 1781859678790380 us; error 27 us; skew 500 ppm
Jun 19 09:01:18 dist-test-slave-mtdr krb5kdc[3430](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859678, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 09:01:18.792652 3805 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 09:01:18.793020 3805 webserver.cc:460] Webserver started at http://127.0.98.3:43367/ using document root <none> and password file <none>
I20260619 09:01:18.793210 3805 fs_manager.cc:359] Metadata directory not provided
I20260619 09:01:18.793267 3805 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 09:01:18.793387 3805 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 09:01:18.794390 3805 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781859658134510-392-0/minicluster-data/ts-2/data/instance:
uuid: "d02a71007efd4437aea53a8652c2b336"
format_stamp: "Formatted at 2026-06-19 09:01:18 on dist-test-slave-mtdr"
server_key: "b9f9a788d88a5de3b49b42e387ae4130"
server_key_iv: "a79f12e33aac658def83845471249220"
server_key_version: "encryptionkey@0"
I20260619 09:01:18.794709 3805 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781859658134510-392-0/minicluster-data/ts-2/wal/instance:
uuid: "d02a71007efd4437aea53a8652c2b336"
format_stamp: "Formatted at 2026-06-19 09:01:18 on dist-test-slave-mtdr"
server_key: "b9f9a788d88a5de3b49b42e387ae4130"
server_key_iv: "a79f12e33aac658def83845471249220"
server_key_version: "encryptionkey@0"
I20260619 09:01:18.795921 3805 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260619 09:01:18.796576 3820 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 09:01:18.796718 3805 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 09:01:18.796795 3805 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781859658134510-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781859658134510-392-0/minicluster-data/ts-2/wal
uuid: "d02a71007efd4437aea53a8652c2b336"
format_stamp: "Formatted at 2026-06-19 09:01:18 on dist-test-slave-mtdr"
server_key: "b9f9a788d88a5de3b49b42e387ae4130"
server_key_iv: "a79f12e33aac658def83845471249220"
server_key_version: "encryptionkey@0"
I20260619 09:01:18.796852 3805 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781859658134510-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781859658134510-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781859658134510-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 09:01:18.822080 3805 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 09:01:18.824210 3805 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 09:01:18.824347 3805 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 09:01:18.824553 3805 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 09:01:18.824820 3805 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 09:01:18.824852 3805 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 09:01:18.824880 3805 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 09:01:18.824904 3805 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 09:01:18.830160 3805 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33905
I20260619 09:01:18.830230 3933 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33905 every 8 connection(s)
I20260619 09:01:18.830503 3805 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781859658134510-392-0/minicluster-data/ts-2/data/info.pb
Jun 19 09:01:18 dist-test-slave-mtdr krb5kdc[3430](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859678, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 09:01:18.838156 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ez5NB/build/release/bin/kudu as pid 3805
I20260619 09:01:18.838284 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781859658134510-392-0/minicluster-data/ts-2/wal/instance
I20260619 09:01:18.838474 392 external_mini_cluster.cc:1428] Setting key 93d38da2f2a077c99eb168c9ad846b1a
I20260619 09:01:18.839212 3531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:18.831185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59861 (local address 127.0.98.62:43509)
0619 09:01:18.831228 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:18.831231 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:18.831433 (+ 202us) server_negotiation.cc:407] Connection header received
0619 09:01:18.831930 (+ 497us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:18.831932 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:18.831943 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:18.831990 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:01:18.832877 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:18.833359 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:18.834306 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:18.834485 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:18.834540 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:18.836584 (+ 2044us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:01:18.836586 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:01:18.836586 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:01:18.836593 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:01:18.837705 (+ 1112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:01:18.838090 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:01:18.838092 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:01:18.838092 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:01:18.838174 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:01:18.838475 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:01:18.838477 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:01:18.838477 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:01:18.838628 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:01:18.838711 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0619 09:01:18.839022 (+ 311us) server_negotiation.cc:299] Negotiation successful
0619 09:01:18.839101 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260619 09:01:18.839421 3936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:18.831157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43509 (local address 127.0.98.3:59861)
0619 09:01:18.831324 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0619 09:01:18.831357 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0619 09:01:18.831840 (+ 483us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:01:18.832072 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:01:18.832076 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:01:18.832599 (+ 523us) client_negotiation.cc:189] Negotiated authn=SASL
0619 09:01:18.832788 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:18.832793 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:18.833434 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:18.833435 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:18.834227 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:18.834229 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:18.834548 (+ 319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:18.834549 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:18.834665 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:18.835117 (+ 452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 09:01:18.835124 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0619 09:01:18.836487 (+ 1363us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 09:01:18.837776 (+ 1289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:01:18.837782 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:01:18.837783 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:01:18.838008 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:01:18.838275 (+ 267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:01:18.838276 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:01:18.838276 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:01:18.838400 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:01:18.838704 (+ 304us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 09:01:18.838710 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 09:01:18.838883 (+ 173us) client_negotiation.cc:769] Sending connection context
0619 09:01:18.839023 (+ 140us) client_negotiation.cc:240] Negotiation successful
0619 09:01:18.839178 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":89,"threads_started":1}
I20260619 09:01:18.839893 3934 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43509
I20260619 09:01:18.840001 3934 heartbeater.cc:464] Registering TS with master...
I20260619 09:01:18.840204 3934 heartbeater.cc:511] Master 127.0.98.62:43509 requested a full tablet report, sending...
I20260619 09:01:18.840508 3475 ts_manager.cc:194] Registered new tserver with Master: d02a71007efd4437aea53a8652c2b336 (127.0.98.3:33905)
I20260619 09:01:18.840837 3475 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:59861
I20260619 09:01:18.850380 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 19 09:01:18 dist-test-slave-mtdr krb5kdc[3430](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859678, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Jun 19 09:01:18 dist-test-slave-mtdr krb5kdc[3430](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859678, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 09:01:18.870993 3531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:18.864388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44922 (local address 127.0.98.62:43509)
0619 09:01:18.864443 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:18.864445 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:18.864469 (+ 24us) server_negotiation.cc:407] Connection header received
0619 09:01:18.864545 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:18.864546 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:18.864556 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:18.864598 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:01:18.865097 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:18.865590 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:18.866277 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:18.866458 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:18.866521 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:18.868617 (+ 2096us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:01:18.868619 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:01:18.868619 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:01:18.868626 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:01:18.869940 (+ 1314us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:01:18.870251 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:01:18.870252 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:01:18.870252 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:01:18.870307 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:01:18.870496 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:01:18.870498 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:01:18.870498 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:01:18.870649 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:01:18.870731 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0619 09:01:18.870849 (+ 118us) server_negotiation.cc:299] Negotiation successful
0619 09:01:18.870927 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
W20260619 09:01:18.871197 3475 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:44922
I20260619 09:01:18.871572 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ez5NB/build/release/bin/kudu with pid 3537
I20260619 09:01:18.875293 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ez5NB/build/release/bin/kudu with pid 3671
I20260619 09:01:18.879254 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ez5NB/build/release/bin/kudu with pid 3805
I20260619 09:01:18.882634 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ez5NB/build/release/bin/kudu with pid 3443
2026-06-19T09:01:18Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (1729 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781859658134510-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 09:01:18 dist-test-slave-mtdr krb5kdc[3946](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 09:01:18 dist-test-slave-mtdr krb5kdc[3946](info): set up 2 sockets
Jun 19 09:01:18 dist-test-slave-mtdr krb5kdc[3946](info): commencing operation
krb5kdc: starting...
W20260619 09:01:19.838282 392 mini_kdc.cc:121] Time spent starting KDC: real 0.937s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781859658134510-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781859658134510-392-0/krb5kdc/test-user.keytab.
Jun 19 09:01:19 dist-test-slave-mtdr krb5kdc[3946](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859679, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T09:01:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T09:01:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781859658134510-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781859658134510-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 09:01:19.946491 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ez5NB/build/release/bin/kudu
/tmp/dist-test-task4ez5NB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781859658134510-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781859658134510-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781859658134510-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781859658134510-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38677
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781859658134510-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45839
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38677
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781859658134510-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781859658134510-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781859658134510-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 09:01:20.024570 3959 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 09:01:20.024777 3959 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 09:01:20.024806 3959 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 09:01:20.025962 3959 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 09:01:20.026003 3959 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 09:01:20.026021 3959 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 09:01:20.026037 3959 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 09:01:20.026052 3959 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 09:01:20.027566 3959 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45839
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781859658134510-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781859658134510-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38677
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781859658134510-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38677
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781859658134510-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3959
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781859658134510-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 09:01:20.027828 3959 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 09:01:20.028031 3959 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 09:01:20.032466 3964 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:01:20.032491 3965 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:01:20.032506 3967 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:01:20.032652 3959 server_base.cc:1034] running on GCE node
I20260619 09:01:20.032859 3959 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 09:01:20.033131 3959 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 09:01:20.034264 3959 hybrid_clock.cc:648] HybridClock initialized: now 1781859680034251 us; error 24 us; skew 500 ppm
Jun 19 09:01:20 dist-test-slave-mtdr krb5kdc[3946](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859680, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 09:01:20.036580 3959 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 09:01:20.037001 3959 webserver.cc:460] Webserver started at http://127.0.98.62:43419/ using document root <none> and password file <none>
I20260619 09:01:20.037189 3959 fs_manager.cc:359] Metadata directory not provided
I20260619 09:01:20.037227 3959 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 09:01:20.037319 3959 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 09:01:20.038301 3959 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781859658134510-392-0/minicluster-data/master-0/data/instance:
uuid: "758dec2a24cc48e08c5d39ad03494ada"
format_stamp: "Formatted at 2026-06-19 09:01:20 on dist-test-slave-mtdr"
server_key: "434022c757d79fa2d8e8a1a9d166e439"
server_key_iv: "82593acc76f889b88ef116a0c150d523"
server_key_version: "encryptionkey@0"
I20260619 09:01:20.038584 3959 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781859658134510-392-0/minicluster-data/master-0/wal/instance:
uuid: "758dec2a24cc48e08c5d39ad03494ada"
format_stamp: "Formatted at 2026-06-19 09:01:20 on dist-test-slave-mtdr"
server_key: "434022c757d79fa2d8e8a1a9d166e439"
server_key_iv: "82593acc76f889b88ef116a0c150d523"
server_key_version: "encryptionkey@0"
I20260619 09:01:20.039829 3959 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 09:01:20.040490 3974 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 09:01:20.040671 3959 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 09:01:20.040750 3959 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781859658134510-392-0/minicluster-data/master-0/data,/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781859658134510-392-0/minicluster-data/master-0/wal
uuid: "758dec2a24cc48e08c5d39ad03494ada"
format_stamp: "Formatted at 2026-06-19 09:01:20 on dist-test-slave-mtdr"
server_key: "434022c757d79fa2d8e8a1a9d166e439"
server_key_iv: "82593acc76f889b88ef116a0c150d523"
server_key_version: "encryptionkey@0"
I20260619 09:01:20.040817 3959 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781859658134510-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781859658134510-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781859658134510-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 09:01:20.059800 3959 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 09:01:20.061856 3959 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 09:01:20.061990 3959 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 09:01:20.065812 3959 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38677
I20260619 09:01:20.065857 4026 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38677 every 8 connection(s)
I20260619 09:01:20.066165 3959 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781859658134510-392-0/minicluster-data/master-0/data/info.pb
I20260619 09:01:20.066987 4027 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 09:01:20.069119 4027 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 758dec2a24cc48e08c5d39ad03494ada: Bootstrap starting.
I20260619 09:01:20.069628 4027 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 758dec2a24cc48e08c5d39ad03494ada: Neither blocks nor log segments found. Creating new log.
I20260619 09:01:20.069869 4027 log.cc:824] T 00000000000000000000000000000000 P 758dec2a24cc48e08c5d39ad03494ada: Log is configured to *not* fsync() on all Append() calls
I20260619 09:01:20.070487 4027 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 758dec2a24cc48e08c5d39ad03494ada: No bootstrap required, opened a new log
I20260619 09:01:20.070657 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ez5NB/build/release/bin/kudu as pid 3959
I20260619 09:01:20.070735 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781859658134510-392-0/minicluster-data/master-0/wal/instance
I20260619 09:01:20.070930 392 external_mini_cluster.cc:1428] Setting key 696a08ed7dfdb588f2c28b83fb4cce13
I20260619 09:01:20.072758 4027 raft_consensus.cc:348] T 00000000000000000000000000000000 P 758dec2a24cc48e08c5d39ad03494ada [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "758dec2a24cc48e08c5d39ad03494ada" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38677 } }
I20260619 09:01:20.072887 4027 raft_consensus.cc:374] T 00000000000000000000000000000000 P 758dec2a24cc48e08c5d39ad03494ada [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 09:01:20.072921 4027 raft_consensus.cc:729] T 00000000000000000000000000000000 P 758dec2a24cc48e08c5d39ad03494ada [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 758dec2a24cc48e08c5d39ad03494ada, State: Initialized, Role: FOLLOWER
I20260619 09:01:20.073048 4027 consensus_queue.cc:260] T 00000000000000000000000000000000 P 758dec2a24cc48e08c5d39ad03494ada [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "758dec2a24cc48e08c5d39ad03494ada" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38677 } }
I20260619 09:01:20.073120 4027 raft_consensus.cc:388] T 00000000000000000000000000000000 P 758dec2a24cc48e08c5d39ad03494ada [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 09:01:20.073149 4027 raft_consensus.cc:482] T 00000000000000000000000000000000 P 758dec2a24cc48e08c5d39ad03494ada [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 09:01:20.073195 4027 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 758dec2a24cc48e08c5d39ad03494ada [term 0 FOLLOWER]: Advancing to term 1
I20260619 09:01:20.073671 4027 raft_consensus.cc:504] T 00000000000000000000000000000000 P 758dec2a24cc48e08c5d39ad03494ada [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "758dec2a24cc48e08c5d39ad03494ada" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38677 } }
I20260619 09:01:20.073758 4027 leader_election.cc:302] T 00000000000000000000000000000000 P 758dec2a24cc48e08c5d39ad03494ada [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 758dec2a24cc48e08c5d39ad03494ada; no voters:
I20260619 09:01:20.073904 4027 leader_election.cc:288] T 00000000000000000000000000000000 P 758dec2a24cc48e08c5d39ad03494ada [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 09:01:20.073944 4032 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 758dec2a24cc48e08c5d39ad03494ada [term 1 FOLLOWER]: Leader election won for term 1
I20260619 09:01:20.074054 4032 raft_consensus.cc:686] T 00000000000000000000000000000000 P 758dec2a24cc48e08c5d39ad03494ada [term 1 LEADER]: Becoming Leader. State: Replica: 758dec2a24cc48e08c5d39ad03494ada, State: Running, Role: LEADER
I20260619 09:01:20.074168 4027 sys_catalog.cc:564] T 00000000000000000000000000000000 P 758dec2a24cc48e08c5d39ad03494ada [sys.catalog]: configured and running, proceeding with master startup.
I20260619 09:01:20.074193 4032 consensus_queue.cc:237] T 00000000000000000000000000000000 P 758dec2a24cc48e08c5d39ad03494ada [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "758dec2a24cc48e08c5d39ad03494ada" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38677 } }
I20260619 09:01:20.074523 4034 sys_catalog.cc:455] T 00000000000000000000000000000000 P 758dec2a24cc48e08c5d39ad03494ada [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "758dec2a24cc48e08c5d39ad03494ada" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "758dec2a24cc48e08c5d39ad03494ada" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38677 } } }
I20260619 09:01:20.074596 4034 sys_catalog.cc:458] T 00000000000000000000000000000000 P 758dec2a24cc48e08c5d39ad03494ada [sys.catalog]: This master's current role is: LEADER
I20260619 09:01:20.074721 4034 sys_catalog.cc:455] T 00000000000000000000000000000000 P 758dec2a24cc48e08c5d39ad03494ada [sys.catalog]: SysCatalogTable state changed. Reason: New leader 758dec2a24cc48e08c5d39ad03494ada. Latest consensus state: current_term: 1 leader_uuid: "758dec2a24cc48e08c5d39ad03494ada" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "758dec2a24cc48e08c5d39ad03494ada" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38677 } } }
I20260619 09:01:20.074764 4034 sys_catalog.cc:458] T 00000000000000000000000000000000 P 758dec2a24cc48e08c5d39ad03494ada [sys.catalog]: This master's current role is: LEADER
I20260619 09:01:20.075102 4039 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 09:01:20.075585 4039 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 09:01:20.077538 4039 catalog_manager.cc:1347] Generated new cluster ID: fb6531cc9ade4782853f644352e3b00e
I20260619 09:01:20.077589 4039 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 19 09:01:20 dist-test-slave-mtdr krb5kdc[3946](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859679, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 09:01:20.081007 4031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:20.071354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51270 (local address 127.0.98.62:38677)
0619 09:01:20.071531 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:20.071538 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:20.071562 (+ 24us) server_negotiation.cc:407] Connection header received
0619 09:01:20.072045 (+ 483us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:20.072052 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:20.072097 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:20.072216 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:01:20.072806 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:20.073779 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:20.074776 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:20.075388 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:20.075497 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:20.078193 (+ 2696us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:01:20.078197 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:01:20.078197 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:01:20.078209 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:01:20.079743 (+ 1534us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:01:20.080046 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:01:20.080049 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:01:20.080049 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:01:20.080130 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:01:20.080291 (+ 161us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:01:20.080292 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:01:20.080292 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:01:20.080519 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:01:20.080661 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0619 09:01:20.080726 (+ 65us) server_negotiation.cc:299] Negotiation successful
0619 09:01:20.080849 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":81,"threads_started":1}
I20260619 09:01:20.082919 4039 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 09:01:20.083570 4039 catalog_manager.cc:1504] Loading token signing keys...
I20260619 09:01:20.091324 4039 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 758dec2a24cc48e08c5d39ad03494ada: Generated new TSK 0
I20260619 09:01:20.091490 4039 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781859658134510-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781859658134510-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 09:01:20.120930 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ez5NB/build/release/bin/kudu
/tmp/dist-test-task4ez5NB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781859658134510-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781859658134510-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781859658134510-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781859658134510-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38677
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781859658134510-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45839
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781859658134510-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781859658134510-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781859658134510-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 09:01:20.234467 4053 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 09:01:20.234742 4053 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 09:01:20.234772 4053 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 09:01:20.236685 4053 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 09:01:20.236755 4053 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 09:01:20.236794 4053 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 09:01:20.238940 4053 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45839
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781859658134510-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781859658134510-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781859658134510-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781859658134510-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38677
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4053
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781859658134510-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 09:01:20.239173 4053 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 09:01:20.239418 4053 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 09:01:20.244657 4053 server_base.cc:1034] running on GCE node
W20260619 09:01:20.244732 4059 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:01:20.244812 4061 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:01:20.244773 4058 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:01:20.245132 4053 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 09:01:20.245380 4053 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 09:01:20.246503 4053 hybrid_clock.cc:648] HybridClock initialized: now 1781859680246496 us; error 28 us; skew 500 ppm
Jun 19 09:01:20 dist-test-slave-mtdr krb5kdc[3946](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859680, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 09:01:20.248893 4053 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 09:01:20.249307 4053 webserver.cc:460] Webserver started at http://127.0.98.1:39641/ using document root <none> and password file <none>
I20260619 09:01:20.249500 4053 fs_manager.cc:359] Metadata directory not provided
I20260619 09:01:20.249545 4053 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 09:01:20.249663 4053 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 09:01:20.250672 4053 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781859658134510-392-0/minicluster-data/ts-0/data/instance:
uuid: "2a261daab10549ac94de816945096165"
format_stamp: "Formatted at 2026-06-19 09:01:20 on dist-test-slave-mtdr"
server_key: "e9bdd8ac2f160d775f21b8b3e2e4bd87"
server_key_iv: "18e117e3312392cdbf06a82b22cd1478"
server_key_version: "encryptionkey@0"
I20260619 09:01:20.251019 4053 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781859658134510-392-0/minicluster-data/ts-0/wal/instance:
uuid: "2a261daab10549ac94de816945096165"
format_stamp: "Formatted at 2026-06-19 09:01:20 on dist-test-slave-mtdr"
server_key: "e9bdd8ac2f160d775f21b8b3e2e4bd87"
server_key_iv: "18e117e3312392cdbf06a82b22cd1478"
server_key_version: "encryptionkey@0"
I20260619 09:01:20.252254 4053 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260619 09:01:20.252903 4068 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 09:01:20.253051 4053 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 09:01:20.253127 4053 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781859658134510-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781859658134510-392-0/minicluster-data/ts-0/wal
uuid: "2a261daab10549ac94de816945096165"
format_stamp: "Formatted at 2026-06-19 09:01:20 on dist-test-slave-mtdr"
server_key: "e9bdd8ac2f160d775f21b8b3e2e4bd87"
server_key_iv: "18e117e3312392cdbf06a82b22cd1478"
server_key_version: "encryptionkey@0"
I20260619 09:01:20.253190 4053 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781859658134510-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781859658134510-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781859658134510-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 09:01:20.274729 4053 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 09:01:20.276708 4053 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 09:01:20.276841 4053 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 09:01:20.277050 4053 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 09:01:20.277340 4053 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 09:01:20.277374 4053 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 09:01:20.277402 4053 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 09:01:20.277422 4053 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 09:01:20.282454 4053 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40507
I20260619 09:01:20.282507 4181 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40507 every 8 connection(s)
I20260619 09:01:20.282790 4053 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781859658134510-392-0/minicluster-data/ts-0/data/info.pb
I20260619 09:01:20.286016 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ez5NB/build/release/bin/kudu as pid 4053
I20260619 09:01:20.286104 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781859658134510-392-0/minicluster-data/ts-0/wal/instance
I20260619 09:01:20.286249 392 external_mini_cluster.cc:1428] Setting key c397f286053c275d750b9299c8ce97ad
Jun 19 09:01:20 dist-test-slave-mtdr krb5kdc[3946](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859680, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 09:01:20.291936 4031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:20.283432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38347 (local address 127.0.98.62:38677)
0619 09:01:20.283488 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:20.283490 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:20.283702 (+ 212us) server_negotiation.cc:407] Connection header received
0619 09:01:20.284587 (+ 885us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:20.284589 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:20.284607 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:20.284658 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:01:20.285245 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:20.285748 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:20.286909 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:20.287141 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:20.287199 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:20.289303 (+ 2104us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:01:20.289305 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:01:20.289305 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:01:20.289313 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:01:20.290705 (+ 1392us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:01:20.291089 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:01:20.291091 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:01:20.291091 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:01:20.291152 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:01:20.291349 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:01:20.291350 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:01:20.291350 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:01:20.291482 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:01:20.291551 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0619 09:01:20.291762 (+ 211us) server_negotiation.cc:299] Negotiation successful
0619 09:01:20.291842 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260619 09:01:20.292009 4184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:20.283429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38677 (local address 127.0.98.1:38347)
0619 09:01:20.283591 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0619 09:01:20.283624 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0619 09:01:20.284503 (+ 879us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:01:20.284734 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:01:20.284737 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:01:20.284969 (+ 232us) client_negotiation.cc:189] Negotiated authn=SASL
0619 09:01:20.285146 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:20.285151 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:20.285827 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:20.285828 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:20.286852 (+ 1024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:20.286854 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:20.287215 (+ 361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:20.287216 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:20.287357 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:20.287853 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 09:01:20.287863 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 09:01:20.289220 (+ 1357us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 09:01:20.290778 (+ 1558us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:01:20.290783 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:01:20.290784 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:01:20.291025 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:01:20.291209 (+ 184us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:01:20.291210 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:01:20.291210 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:01:20.291297 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:01:20.291540 (+ 243us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 09:01:20.291543 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 09:01:20.291653 (+ 110us) client_negotiation.cc:769] Sending connection context
0619 09:01:20.291735 (+ 82us) client_negotiation.cc:240] Negotiation successful
0619 09:01:20.291847 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":92,"threads_started":1}
I20260619 09:01:20.292445 4182 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38677
I20260619 09:01:20.292529 4182 heartbeater.cc:464] Registering TS with master...
I20260619 09:01:20.292687 4182 heartbeater.cc:511] Master 127.0.98.62:38677 requested a full tablet report, sending...
I20260619 09:01:20.293228 3982 ts_manager.cc:194] Registered new tserver with Master: 2a261daab10549ac94de816945096165 (127.0.98.1:40507)
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260619 09:01:20.294148 3982 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:38347
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781859658134510-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781859658134510-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 09:01:20.309990 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ez5NB/build/release/bin/kudu
/tmp/dist-test-task4ez5NB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781859658134510-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781859658134510-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781859658134510-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781859658134510-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38677
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781859658134510-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45839
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781859658134510-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781859658134510-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781859658134510-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 09:01:20.386749 4187 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 09:01:20.386966 4187 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 09:01:20.386998 4187 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 09:01:20.388119 4187 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 09:01:20.388156 4187 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 09:01:20.388190 4187 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 09:01:20.389585 4187 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45839
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781859658134510-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781859658134510-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781859658134510-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781859658134510-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38677
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4187
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781859658134510-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 09:01:20.389798 4187 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 09:01:20.389981 4187 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 09:01:20.394510 4193 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:01:20.394548 4195 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:01:20.394558 4192 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:01:20.394944 4187 server_base.cc:1034] running on GCE node
I20260619 09:01:20.395124 4187 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 09:01:20.395330 4187 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 09:01:20.396462 4187 hybrid_clock.cc:648] HybridClock initialized: now 1781859680396448 us; error 31 us; skew 500 ppm
Jun 19 09:01:20 dist-test-slave-mtdr krb5kdc[3946](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859680, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 09:01:20.401429 4187 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 09:01:20.401840 4187 webserver.cc:460] Webserver started at http://127.0.98.2:35869/ using document root <none> and password file <none>
I20260619 09:01:20.402019 4187 fs_manager.cc:359] Metadata directory not provided
I20260619 09:01:20.402060 4187 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 09:01:20.402179 4187 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 09:01:20.403191 4187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781859658134510-392-0/minicluster-data/ts-1/data/instance:
uuid: "c4d274c96cdf47679f16e85264ed6d6f"
format_stamp: "Formatted at 2026-06-19 09:01:20 on dist-test-slave-mtdr"
server_key: "b304c01c392dc3cff38dde73f9211947"
server_key_iv: "317f4a42100d50cd671282e2ee84a3e8"
server_key_version: "encryptionkey@0"
I20260619 09:01:20.403487 4187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781859658134510-392-0/minicluster-data/ts-1/wal/instance:
uuid: "c4d274c96cdf47679f16e85264ed6d6f"
format_stamp: "Formatted at 2026-06-19 09:01:20 on dist-test-slave-mtdr"
server_key: "b304c01c392dc3cff38dde73f9211947"
server_key_iv: "317f4a42100d50cd671282e2ee84a3e8"
server_key_version: "encryptionkey@0"
I20260619 09:01:20.404610 4187 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 09:01:20.405263 4202 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 09:01:20.405434 4187 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 09:01:20.405510 4187 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781859658134510-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781859658134510-392-0/minicluster-data/ts-1/wal
uuid: "c4d274c96cdf47679f16e85264ed6d6f"
format_stamp: "Formatted at 2026-06-19 09:01:20 on dist-test-slave-mtdr"
server_key: "b304c01c392dc3cff38dde73f9211947"
server_key_iv: "317f4a42100d50cd671282e2ee84a3e8"
server_key_version: "encryptionkey@0"
I20260619 09:01:20.405558 4187 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781859658134510-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781859658134510-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781859658134510-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 09:01:20.415977 4187 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 09:01:20.417793 4187 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 09:01:20.417896 4187 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 09:01:20.418051 4187 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 09:01:20.418277 4187 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 09:01:20.418314 4187 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 09:01:20.418334 4187 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 09:01:20.418354 4187 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 09:01:20.423437 4187 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36813
I20260619 09:01:20.423511 4315 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36813 every 8 connection(s)
I20260619 09:01:20.423767 4187 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781859658134510-392-0/minicluster-data/ts-1/data/info.pb
I20260619 09:01:20.424041 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ez5NB/build/release/bin/kudu as pid 4187
I20260619 09:01:20.424132 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781859658134510-392-0/minicluster-data/ts-1/wal/instance
I20260619 09:01:20.424316 392 external_mini_cluster.cc:1428] Setting key 992eea361307e9e5d9a7f459d30b336d
Jun 19 09:01:20 dist-test-slave-mtdr krb5kdc[3946](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859680, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260619 09:01:20.433809 4031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:20.424482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34665 (local address 127.0.98.62:38677)
0619 09:01:20.424555 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:20.424557 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:20.424771 (+ 214us) server_negotiation.cc:407] Connection header received
0619 09:01:20.425896 (+ 1125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:20.425897 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:20.425908 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:20.425958 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:01:20.426677 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:20.427427 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:20.428794 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:20.428983 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:20.429045 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:20.431307 (+ 2262us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:01:20.431308 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:01:20.431309 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:01:20.431315 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:01:20.432697 (+ 1382us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:01:20.433023 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:01:20.433024 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:01:20.433024 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:01:20.433072 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:01:20.433293 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:01:20.433294 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:01:20.433294 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:01:20.433411 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:01:20.433471 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0619 09:01:20.433675 (+ 204us) server_negotiation.cc:299] Negotiation successful
0619 09:01:20.433742 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260619 09:01:20.433930 4318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:20.424474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38677 (local address 127.0.98.2:34665)
0619 09:01:20.424659 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0619 09:01:20.424702 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 09:01:20.425780 (+ 1078us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:01:20.426032 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:01:20.426037 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:01:20.426364 (+ 327us) client_negotiation.cc:189] Negotiated authn=SASL
0619 09:01:20.426590 (+ 226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:20.426594 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:20.427515 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:20.427516 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:20.428718 (+ 1202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:20.428720 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:20.429062 (+ 342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:20.429063 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:20.429196 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:20.429672 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 09:01:20.429682 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 09:01:20.431213 (+ 1531us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 09:01:20.432766 (+ 1553us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:01:20.432771 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:01:20.432773 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:01:20.432966 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:01:20.433118 (+ 152us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:01:20.433118 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:01:20.433119 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:01:20.433227 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:01:20.433466 (+ 239us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 09:01:20.433469 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 09:01:20.433571 (+ 102us) client_negotiation.cc:769] Sending connection context
0619 09:01:20.433646 (+ 75us) client_negotiation.cc:240] Negotiation successful
0619 09:01:20.433751 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":84,"threads_started":1}
I20260619 09:01:20.434337 4316 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38677
I20260619 09:01:20.434428 4316 heartbeater.cc:464] Registering TS with master...
I20260619 09:01:20.434602 4316 heartbeater.cc:511] Master 127.0.98.62:38677 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260619 09:01:20.434996 3982 ts_manager.cc:194] Registered new tserver with Master: c4d274c96cdf47679f16e85264ed6d6f (127.0.98.2:36813)
I20260619 09:01:20.435344 3982 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:34665
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781859658134510-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781859658134510-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 09:01:20.447512 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ez5NB/build/release/bin/kudu
/tmp/dist-test-task4ez5NB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781859658134510-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781859658134510-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781859658134510-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781859658134510-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38677
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781859658134510-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45839
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781859658134510-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781859658134510-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781859658134510-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 09:01:20.524305 4321 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 09:01:20.524502 4321 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 09:01:20.524530 4321 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 09:01:20.525635 4321 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 09:01:20.525671 4321 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 09:01:20.525705 4321 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 09:01:20.527130 4321 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45839
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781859658134510-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781859658134510-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781859658134510-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781859658134510-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38677
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4321
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781859658134510-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 09:01:20.527352 4321 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 09:01:20.527544 4321 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 09:01:20.532047 4327 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:01:20.532169 4321 server_base.cc:1034] running on GCE node
W20260619 09:01:20.532047 4326 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:01:20.532083 4329 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:01:20.532454 4321 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 09:01:20.532665 4321 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 09:01:20.533836 4321 hybrid_clock.cc:648] HybridClock initialized: now 1781859680533816 us; error 28 us; skew 500 ppm
Jun 19 09:01:20 dist-test-slave-mtdr krb5kdc[3946](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859680, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 09:01:20.536386 4321 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 09:01:20.536760 4321 webserver.cc:460] Webserver started at http://127.0.98.3:33057/ using document root <none> and password file <none>
I20260619 09:01:20.536931 4321 fs_manager.cc:359] Metadata directory not provided
I20260619 09:01:20.536975 4321 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 09:01:20.537060 4321 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 09:01:20.538087 4321 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781859658134510-392-0/minicluster-data/ts-2/data/instance:
uuid: "26875dbcbc8d48c38dff96ad5bdce0e6"
format_stamp: "Formatted at 2026-06-19 09:01:20 on dist-test-slave-mtdr"
server_key: "ae419cddfb414a456f671739097996a5"
server_key_iv: "ceb19cf4d622ff95ea7086982b967bac"
server_key_version: "encryptionkey@0"
I20260619 09:01:20.538448 4321 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781859658134510-392-0/minicluster-data/ts-2/wal/instance:
uuid: "26875dbcbc8d48c38dff96ad5bdce0e6"
format_stamp: "Formatted at 2026-06-19 09:01:20 on dist-test-slave-mtdr"
server_key: "ae419cddfb414a456f671739097996a5"
server_key_iv: "ceb19cf4d622ff95ea7086982b967bac"
server_key_version: "encryptionkey@0"
I20260619 09:01:20.539727 4321 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 09:01:20.540328 4336 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 09:01:20.540488 4321 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 09:01:20.540565 4321 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781859658134510-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781859658134510-392-0/minicluster-data/ts-2/wal
uuid: "26875dbcbc8d48c38dff96ad5bdce0e6"
format_stamp: "Formatted at 2026-06-19 09:01:20 on dist-test-slave-mtdr"
server_key: "ae419cddfb414a456f671739097996a5"
server_key_iv: "ceb19cf4d622ff95ea7086982b967bac"
server_key_version: "encryptionkey@0"
I20260619 09:01:20.540613 4321 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781859658134510-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781859658134510-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781859658134510-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 09:01:20.548141 4321 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 09:01:20.550076 4321 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 09:01:20.550175 4321 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 09:01:20.550340 4321 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 09:01:20.550558 4321 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 09:01:20.550590 4321 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 09:01:20.550616 4321 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 09:01:20.550631 4321 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 09:01:20.555832 4321 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35225
I20260619 09:01:20.555883 4449 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35225 every 8 connection(s)
I20260619 09:01:20.556167 4321 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781859658134510-392-0/minicluster-data/ts-2/data/info.pb
I20260619 09:01:20.561594 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ez5NB/build/release/bin/kudu as pid 4321
I20260619 09:01:20.561692 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781859658134510-392-0/minicluster-data/ts-2/wal/instance
I20260619 09:01:20.561857 392 external_mini_cluster.cc:1428] Setting key 846bb6f7d16b606f454d3d132353bc8f
Jun 19 09:01:20 dist-test-slave-mtdr krb5kdc[3946](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859680, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 09:01:20.564781 4031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:20.556802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50493 (local address 127.0.98.62:38677)
0619 09:01:20.556844 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:20.556846 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:20.557042 (+ 196us) server_negotiation.cc:407] Connection header received
0619 09:01:20.557964 (+ 922us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:20.557966 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:20.557977 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:20.558026 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:01:20.558613 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:20.559197 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:20.560129 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:20.560325 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:20.560379 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:20.562342 (+ 1963us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:01:20.562343 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:01:20.562344 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:01:20.562351 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:01:20.563602 (+ 1251us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:01:20.563945 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:01:20.563946 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:01:20.563946 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:01:20.564002 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:01:20.564205 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:01:20.564206 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:01:20.564206 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:01:20.564316 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:01:20.564373 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0619 09:01:20.564617 (+ 244us) server_negotiation.cc:299] Negotiation successful
0619 09:01:20.564692 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260619 09:01:20.564939 4452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:20.556775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38677 (local address 127.0.98.3:50493)
0619 09:01:20.556927 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0619 09:01:20.556964 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0619 09:01:20.557873 (+ 909us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:01:20.558089 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:01:20.558093 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:01:20.558333 (+ 240us) client_negotiation.cc:189] Negotiated authn=SASL
0619 09:01:20.558535 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:20.558539 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:20.559271 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:20.559272 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:20.560052 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:20.560053 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:20.560388 (+ 335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:20.560389 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:20.560502 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:20.560921 (+ 419us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 09:01:20.560928 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0619 09:01:20.562266 (+ 1338us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 09:01:20.563673 (+ 1407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:01:20.563678 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:01:20.563680 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:01:20.563882 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:01:20.564053 (+ 171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:01:20.564053 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:01:20.564054 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:01:20.564153 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:01:20.564368 (+ 215us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 09:01:20.564374 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 09:01:20.564508 (+ 134us) client_negotiation.cc:769] Sending connection context
0619 09:01:20.564592 (+ 84us) client_negotiation.cc:240] Negotiation successful
0619 09:01:20.564735 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":71,"threads_started":1}
I20260619 09:01:20.565320 4450 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38677
I20260619 09:01:20.565393 4450 heartbeater.cc:464] Registering TS with master...
I20260619 09:01:20.565548 4450 heartbeater.cc:511] Master 127.0.98.62:38677 requested a full tablet report, sending...
I20260619 09:01:20.565850 3982 ts_manager.cc:194] Registered new tserver with Master: 26875dbcbc8d48c38dff96ad5bdce0e6 (127.0.98.3:35225)
I20260619 09:01:20.566200 3982 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:50493
I20260619 09:01:20.573382 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 19 09:01:20 dist-test-slave-mtdr krb5kdc[3946](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859680, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 09:01:20 dist-test-slave-mtdr krb5kdc[3946](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859680, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260619 09:01:20.595029 4457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:20.588013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44052 (local address 127.0.98.1:40507)
0619 09:01:20.588170 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:20.588174 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:20.588185 (+ 11us) server_negotiation.cc:407] Connection header received
0619 09:01:20.588210 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:20.588211 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:20.588225 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:20.588294 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:01:20.588787 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:20.589493 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:20.590183 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:20.590523 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:20.590579 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:20.592691 (+ 2112us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:01:20.592698 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:01:20.592698 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:01:20.592704 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:01:20.594003 (+ 1299us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:01:20.594300 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:01:20.594303 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:01:20.594303 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:01:20.594361 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:01:20.594490 (+ 129us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:01:20.594490 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:01:20.594491 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:01:20.594611 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:01:20.594719 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0619 09:01:20.594888 (+ 169us) server_negotiation.cc:299] Negotiation successful
0619 09:01:20.594951 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":75,"threads_started":1}
Jun 19 09:01:20 dist-test-slave-mtdr krb5kdc[3946](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859680, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 09:01:20.602242 4031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:20.596267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51284 (local address 127.0.98.62:38677)
0619 09:01:20.596315 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:20.596316 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:20.596326 (+ 10us) server_negotiation.cc:407] Connection header received
0619 09:01:20.596380 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:20.596381 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:20.596388 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:20.596432 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:01:20.596814 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:20.597313 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:20.597931 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:20.598114 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:20.598166 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:20.600061 (+ 1895us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:01:20.600062 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:01:20.600062 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:01:20.600069 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:01:20.601308 (+ 1239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:01:20.601637 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:01:20.601638 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:01:20.601638 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:01:20.601685 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:01:20.601821 (+ 136us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:01:20.601821 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:01:20.601821 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:01:20.601944 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:01:20.602009 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0619 09:01:20.602125 (+ 116us) server_negotiation.cc:299] Negotiation successful
0619 09:01:20.602179 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
W20260619 09:01:20.602474 3982 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51284
I20260619 09:01:20.602741 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ez5NB/build/release/bin/kudu with pid 4053
I20260619 09:01:20.606521 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ez5NB/build/release/bin/kudu with pid 4187
I20260619 09:01:20.610307 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ez5NB/build/release/bin/kudu with pid 4321
I20260619 09:01:20.613785 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ez5NB/build/release/bin/kudu with pid 3959
2026-06-19T09:01:20Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (1731 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781859658134510-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 09:01:20 dist-test-slave-mtdr krb5kdc[4462](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 09:01:20 dist-test-slave-mtdr krb5kdc[4462](info): set up 2 sockets
Jun 19 09:01:20 dist-test-slave-mtdr krb5kdc[4462](info): commencing operation
krb5kdc: starting...
W20260619 09:01:21.570554 392 mini_kdc.cc:121] Time spent starting KDC: real 0.938s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781859658134510-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781859658134510-392-0/krb5kdc/test-user.keytab.
Jun 19 09:01:21 dist-test-slave-mtdr krb5kdc[4462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859681, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T09:01:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T09:01:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781859658134510-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781859658134510-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 09:01:21.675660 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ez5NB/build/release/bin/kudu
/tmp/dist-test-task4ez5NB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781859658134510-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781859658134510-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781859658134510-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781859658134510-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33019
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781859658134510-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33019
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781859658134510-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781859658134510-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781859658134510-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 09:01:21.752099 4475 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 09:01:21.752286 4475 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 09:01:21.752314 4475 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 09:01:21.753409 4475 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 09:01:21.753446 4475 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 09:01:21.753464 4475 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 09:01:21.753479 4475 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 09:01:21.753494 4475 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 09:01:21.754982 4475 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781859658134510-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781859658134510-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33019
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781859658134510-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33019
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781859658134510-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4475
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781859658134510-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 09:01:21.755210 4475 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 09:01:21.755407 4475 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 09:01:21.759812 4481 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:01:21.759977 4483 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:01:21.760466 4480 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:01:21.760520 4475 server_base.cc:1034] running on GCE node
I20260619 09:01:21.760792 4475 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 09:01:21.761130 4475 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 09:01:21.762288 4475 hybrid_clock.cc:648] HybridClock initialized: now 1781859681762271 us; error 37 us; skew 500 ppm
Jun 19 09:01:21 dist-test-slave-mtdr krb5kdc[4462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859681, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 09:01:21.765363 4475 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 09:01:21.765914 4475 fs_manager.cc:359] Metadata directory not provided
I20260619 09:01:21.765988 4475 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 09:01:21.766124 4475 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 09:01:21.767699 4475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781859658134510-392-0/minicluster-data/master-0/data/instance:
uuid: "aa813b46e3904b58bac3fd579acd3e81"
format_stamp: "Formatted at 2026-06-19 09:01:21 on dist-test-slave-mtdr"
server_key: "aa7556040722478a4d2c01a8e790a16b"
server_key_iv: "94827c0962783c70135611eda9da03cb"
server_key_version: "encryptionkey@0"
I20260619 09:01:21.768076 4475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781859658134510-392-0/minicluster-data/master-0/wal/instance:
uuid: "aa813b46e3904b58bac3fd579acd3e81"
format_stamp: "Formatted at 2026-06-19 09:01:21 on dist-test-slave-mtdr"
server_key: "aa7556040722478a4d2c01a8e790a16b"
server_key_iv: "94827c0962783c70135611eda9da03cb"
server_key_version: "encryptionkey@0"
I20260619 09:01:21.769943 4475 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260619 09:01:21.770893 4489 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 09:01:21.771234 4475 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 09:01:21.771330 4475 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781859658134510-392-0/minicluster-data/master-0/data,/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781859658134510-392-0/minicluster-data/master-0/wal
uuid: "aa813b46e3904b58bac3fd579acd3e81"
format_stamp: "Formatted at 2026-06-19 09:01:21 on dist-test-slave-mtdr"
server_key: "aa7556040722478a4d2c01a8e790a16b"
server_key_iv: "94827c0962783c70135611eda9da03cb"
server_key_version: "encryptionkey@0"
I20260619 09:01:21.771404 4475 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781859658134510-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781859658134510-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781859658134510-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 09:01:21.792881 4475 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 09:01:21.795322 4475 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 09:01:21.795488 4475 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 09:01:21.800683 4475 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33019
I20260619 09:01:21.800704 4541 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33019 every 8 connection(s)
I20260619 09:01:21.801041 4475 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781859658134510-392-0/minicluster-data/master-0/data/info.pb
I20260619 09:01:21.802073 4542 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 09:01:21.805253 4542 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa813b46e3904b58bac3fd579acd3e81: Bootstrap starting.
I20260619 09:01:21.805974 4542 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aa813b46e3904b58bac3fd579acd3e81: Neither blocks nor log segments found. Creating new log.
I20260619 09:01:21.806321 4542 log.cc:824] T 00000000000000000000000000000000 P aa813b46e3904b58bac3fd579acd3e81: Log is configured to *not* fsync() on all Append() calls
I20260619 09:01:21.807061 4542 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa813b46e3904b58bac3fd579acd3e81: No bootstrap required, opened a new log
I20260619 09:01:21.809233 4542 raft_consensus.cc:348] T 00000000000000000000000000000000 P aa813b46e3904b58bac3fd579acd3e81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa813b46e3904b58bac3fd579acd3e81" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33019 } }
I20260619 09:01:21.809427 4542 raft_consensus.cc:374] T 00000000000000000000000000000000 P aa813b46e3904b58bac3fd579acd3e81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 09:01:21.809479 4542 raft_consensus.cc:729] T 00000000000000000000000000000000 P aa813b46e3904b58bac3fd579acd3e81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa813b46e3904b58bac3fd579acd3e81, State: Initialized, Role: FOLLOWER
I20260619 09:01:21.809607 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ez5NB/build/release/bin/kudu as pid 4475
I20260619 09:01:21.809696 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781859658134510-392-0/minicluster-data/master-0/wal/instance
I20260619 09:01:21.809638 4542 consensus_queue.cc:260] T 00000000000000000000000000000000 P aa813b46e3904b58bac3fd579acd3e81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa813b46e3904b58bac3fd579acd3e81" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33019 } }
I20260619 09:01:21.809732 4542 raft_consensus.cc:388] T 00000000000000000000000000000000 P aa813b46e3904b58bac3fd579acd3e81 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 09:01:21.809760 4542 raft_consensus.cc:482] T 00000000000000000000000000000000 P aa813b46e3904b58bac3fd579acd3e81 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 09:01:21.809808 4542 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aa813b46e3904b58bac3fd579acd3e81 [term 0 FOLLOWER]: Advancing to term 1
I20260619 09:01:21.809871 392 external_mini_cluster.cc:1428] Setting key 805f7c2e2d086da067062b82cdba8b41
I20260619 09:01:21.810828 4542 raft_consensus.cc:504] T 00000000000000000000000000000000 P aa813b46e3904b58bac3fd579acd3e81 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa813b46e3904b58bac3fd579acd3e81" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33019 } }
I20260619 09:01:21.810940 4542 leader_election.cc:302] T 00000000000000000000000000000000 P aa813b46e3904b58bac3fd579acd3e81 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aa813b46e3904b58bac3fd579acd3e81; no voters:
I20260619 09:01:21.811102 4542 leader_election.cc:288] T 00000000000000000000000000000000 P aa813b46e3904b58bac3fd579acd3e81 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 09:01:21.811124 4547 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aa813b46e3904b58bac3fd579acd3e81 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 09:01:21.811291 4547 raft_consensus.cc:686] T 00000000000000000000000000000000 P aa813b46e3904b58bac3fd579acd3e81 [term 1 LEADER]: Becoming Leader. State: Replica: aa813b46e3904b58bac3fd579acd3e81, State: Running, Role: LEADER
I20260619 09:01:21.811345 4542 sys_catalog.cc:564] T 00000000000000000000000000000000 P aa813b46e3904b58bac3fd579acd3e81 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 09:01:21.811398 4547 consensus_queue.cc:237] T 00000000000000000000000000000000 P aa813b46e3904b58bac3fd579acd3e81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa813b46e3904b58bac3fd579acd3e81" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33019 } }
I20260619 09:01:21.811681 4547 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa813b46e3904b58bac3fd579acd3e81 [sys.catalog]: SysCatalogTable state changed. Reason: New leader aa813b46e3904b58bac3fd579acd3e81. Latest consensus state: current_term: 1 leader_uuid: "aa813b46e3904b58bac3fd579acd3e81" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa813b46e3904b58bac3fd579acd3e81" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33019 } } }
I20260619 09:01:21.811753 4547 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa813b46e3904b58bac3fd579acd3e81 [sys.catalog]: This master's current role is: LEADER
W20260619 09:01:21.813035 4563 catalog_manager.cc:1558] T 00000000000000000000000000000000 P aa813b46e3904b58bac3fd579acd3e81: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 09:01:21.813091 4563 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 09:01:21.813133 4554 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 09:01:21.813280 4554 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 09:01:21.814718 4554 catalog_manager.cc:1347] Generated new cluster ID: eea25e6052c045468664742e6ce274ad
I20260619 09:01:21.814764 4554 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 19 09:01:21 dist-test-slave-mtdr krb5kdc[4462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859681, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 09:01:21.818241 4549 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa813b46e3904b58bac3fd579acd3e81 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "aa813b46e3904b58bac3fd579acd3e81" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa813b46e3904b58bac3fd579acd3e81" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33019 } } }
I20260619 09:01:21.818370 4549 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa813b46e3904b58bac3fd579acd3e81 [sys.catalog]: This master's current role is: LEADER
I20260619 09:01:21.823613 4546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:21.810306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40948 (local address 127.0.98.62:33019)
0619 09:01:21.810505 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:21.810513 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:21.810539 (+ 26us) server_negotiation.cc:407] Connection header received
0619 09:01:21.810630 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:21.810636 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:21.810677 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:21.810805 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:01:21.811635 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:21.813553 (+ 1918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:21.814564 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:21.815281 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:21.815661 (+ 380us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:21.817933 (+ 2272us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:01:21.817937 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:01:21.817937 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:01:21.817948 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:01:21.821536 (+ 3588us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:01:21.821898 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:01:21.821901 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:01:21.821902 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:01:21.821996 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:01:21.822224 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:01:21.822225 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:01:21.822225 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:01:21.822553 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:01:21.823130 (+ 577us) server_negotiation.cc:1035] Waiting for connection context
0619 09:01:21.823224 (+ 94us) server_negotiation.cc:299] Negotiation successful
0619 09:01:21.823377 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":87,"threads_started":1}
I20260619 09:01:21.842089 4554 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 09:01:21.842638 4554 catalog_manager.cc:1504] Loading token signing keys...
I20260619 09:01:21.848500 4554 catalog_manager.cc:5868] T 00000000000000000000000000000000 P aa813b46e3904b58bac3fd579acd3e81: Generated new TSK 0
I20260619 09:01:21.848667 4554 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781859658134510-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781859658134510-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 09:01:21.874995 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ez5NB/build/release/bin/kudu
/tmp/dist-test-task4ez5NB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781859658134510-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781859658134510-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781859658134510-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781859658134510-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33019
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781859658134510-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781859658134510-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781859658134510-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781859658134510-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 09:01:21.951988 4568 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 09:01:21.952219 4568 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 09:01:21.952252 4568 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 09:01:21.953368 4568 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 09:01:21.953404 4568 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 09:01:21.953438 4568 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 09:01:21.954895 4568 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781859658134510-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781859658134510-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781859658134510-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781859658134510-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33019
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4568
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781859658134510-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 09:01:21.955117 4568 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 09:01:21.955307 4568 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 09:01:21.959838 4576 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:01:21.959869 4568 server_base.cc:1034] running on GCE node
W20260619 09:01:21.959890 4574 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:01:21.959843 4573 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:01:21.960187 4568 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 09:01:21.960403 4568 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 09:01:21.961555 4568 hybrid_clock.cc:648] HybridClock initialized: now 1781859681961542 us; error 26 us; skew 500 ppm
Jun 19 09:01:21 dist-test-slave-mtdr krb5kdc[4462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859681, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 09:01:21.963996 4568 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 09:01:21.964304 4568 fs_manager.cc:359] Metadata directory not provided
I20260619 09:01:21.964354 4568 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 09:01:21.964470 4568 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 09:01:21.965510 4568 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781859658134510-392-0/minicluster-data/ts-0/data/instance:
uuid: "3c1f48243a8c4435915b694698b999c5"
format_stamp: "Formatted at 2026-06-19 09:01:21 on dist-test-slave-mtdr"
server_key: "1a6dfdf1ba21779d32c77de12a3a52fa"
server_key_iv: "8dfd0a114b1c0b05330434fcdb41cc89"
server_key_version: "encryptionkey@0"
I20260619 09:01:21.965832 4568 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781859658134510-392-0/minicluster-data/ts-0/wal/instance:
uuid: "3c1f48243a8c4435915b694698b999c5"
format_stamp: "Formatted at 2026-06-19 09:01:21 on dist-test-slave-mtdr"
server_key: "1a6dfdf1ba21779d32c77de12a3a52fa"
server_key_iv: "8dfd0a114b1c0b05330434fcdb41cc89"
server_key_version: "encryptionkey@0"
I20260619 09:01:21.967013 4568 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 09:01:21.967685 4582 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 09:01:21.967847 4568 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 09:01:21.967923 4568 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781859658134510-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781859658134510-392-0/minicluster-data/ts-0/wal
uuid: "3c1f48243a8c4435915b694698b999c5"
format_stamp: "Formatted at 2026-06-19 09:01:21 on dist-test-slave-mtdr"
server_key: "1a6dfdf1ba21779d32c77de12a3a52fa"
server_key_iv: "8dfd0a114b1c0b05330434fcdb41cc89"
server_key_version: "encryptionkey@0"
I20260619 09:01:21.967978 4568 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781859658134510-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781859658134510-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781859658134510-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 09:01:21.978421 4568 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 09:01:21.980309 4568 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 09:01:21.980436 4568 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 09:01:21.980623 4568 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 09:01:21.980887 4568 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 09:01:21.980919 4568 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 09:01:21.980947 4568 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 09:01:21.980964 4568 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 09:01:21.986296 4568 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38021
I20260619 09:01:21.986349 4695 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38021 every 8 connection(s)
I20260619 09:01:21.986567 4568 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781859658134510-392-0/minicluster-data/ts-0/data/info.pb
I20260619 09:01:21.989459 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ez5NB/build/release/bin/kudu as pid 4568
I20260619 09:01:21.989574 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781859658134510-392-0/minicluster-data/ts-0/wal/instance
I20260619 09:01:21.989743 392 external_mini_cluster.cc:1428] Setting key 3047d7db900b5db718ed57cb001078d0
Jun 19 09:01:21 dist-test-slave-mtdr krb5kdc[4462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859681, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 09:01:21.995232 4546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:21.987248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48343 (local address 127.0.98.62:33019)
0619 09:01:21.987316 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:21.987318 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:21.987478 (+ 160us) server_negotiation.cc:407] Connection header received
0619 09:01:21.987937 (+ 459us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:21.987939 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:21.987953 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:21.988002 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:01:21.988885 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:21.989377 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:21.990407 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:21.990615 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:21.990669 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:21.992787 (+ 2118us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:01:21.992788 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:01:21.992788 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:01:21.992798 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:01:21.994006 (+ 1208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:01:21.994347 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:01:21.994348 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:01:21.994349 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:01:21.994401 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:01:21.994605 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:01:21.994606 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:01:21.994606 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:01:21.994711 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:01:21.994773 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0619 09:01:21.995048 (+ 275us) server_negotiation.cc:299] Negotiation successful
0619 09:01:21.995125 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260619 09:01:21.995371 4698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:21.987214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33019 (local address 127.0.98.1:48343)
0619 09:01:21.987370 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0619 09:01:21.987402 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0619 09:01:21.987844 (+ 442us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:01:21.988063 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:01:21.988066 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:01:21.988311 (+ 245us) client_negotiation.cc:189] Negotiated authn=SASL
0619 09:01:21.988770 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:21.988775 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:21.989456 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:21.989457 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:21.990324 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:21.990327 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:21.990679 (+ 352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:21.990680 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:21.990797 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:21.991277 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 09:01:21.991287 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 09:01:21.992699 (+ 1412us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 09:01:21.994076 (+ 1377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:01:21.994082 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:01:21.994084 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:01:21.994285 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:01:21.994453 (+ 168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:01:21.994454 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:01:21.994454 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:01:21.994553 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:01:21.994764 (+ 211us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 09:01:21.994769 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 09:01:21.994935 (+ 166us) client_negotiation.cc:769] Sending connection context
0619 09:01:21.995021 (+ 86us) client_negotiation.cc:240] Negotiation successful
0619 09:01:21.995160 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":84,"threads_started":1}
I20260619 09:01:21.995841 4696 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33019
I20260619 09:01:21.995942 4696 heartbeater.cc:464] Registering TS with master...
I20260619 09:01:21.996109 4696 heartbeater.cc:511] Master 127.0.98.62:33019 requested a full tablet report, sending...
I20260619 09:01:21.996587 4506 ts_manager.cc:194] Registered new tserver with Master: 3c1f48243a8c4435915b694698b999c5 (127.0.98.1:38021)
I20260619 09:01:21.997159 4506 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:48343
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781859658134510-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781859658134510-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 09:01:22.013477 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ez5NB/build/release/bin/kudu
/tmp/dist-test-task4ez5NB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781859658134510-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781859658134510-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781859658134510-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781859658134510-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33019
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781859658134510-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781859658134510-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781859658134510-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781859658134510-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 09:01:22.090173 4701 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 09:01:22.090364 4701 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 09:01:22.090386 4701 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 09:01:22.091493 4701 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 09:01:22.091526 4701 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 09:01:22.091552 4701 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 09:01:22.092957 4701 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781859658134510-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781859658134510-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781859658134510-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781859658134510-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33019
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4701
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781859658134510-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 09:01:22.093127 4701 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 09:01:22.093295 4701 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 09:01:22.097743 4707 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:01:22.097743 4709 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:01:22.097759 4706 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:01:22.098132 4701 server_base.cc:1034] running on GCE node
I20260619 09:01:22.098315 4701 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 09:01:22.098537 4701 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 09:01:22.099669 4701 hybrid_clock.cc:648] HybridClock initialized: now 1781859682099658 us; error 31 us; skew 500 ppm
Jun 19 09:01:22 dist-test-slave-mtdr krb5kdc[4462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859682, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 09:01:22.101992 4701 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 09:01:22.102316 4701 fs_manager.cc:359] Metadata directory not provided
I20260619 09:01:22.102370 4701 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 09:01:22.102492 4701 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 09:01:22.103534 4701 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781859658134510-392-0/minicluster-data/ts-1/data/instance:
uuid: "f1a9826769b54db089ba41c6e9fde0f5"
format_stamp: "Formatted at 2026-06-19 09:01:22 on dist-test-slave-mtdr"
server_key: "f58b6b669807656dc678edf00cc800fc"
server_key_iv: "214b5d756e480565ec06a21fa0806246"
server_key_version: "encryptionkey@0"
I20260619 09:01:22.103855 4701 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781859658134510-392-0/minicluster-data/ts-1/wal/instance:
uuid: "f1a9826769b54db089ba41c6e9fde0f5"
format_stamp: "Formatted at 2026-06-19 09:01:22 on dist-test-slave-mtdr"
server_key: "f58b6b669807656dc678edf00cc800fc"
server_key_iv: "214b5d756e480565ec06a21fa0806246"
server_key_version: "encryptionkey@0"
I20260619 09:01:22.105015 4701 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 09:01:22.105783 4715 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 09:01:22.105940 4701 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 09:01:22.106025 4701 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781859658134510-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781859658134510-392-0/minicluster-data/ts-1/wal
uuid: "f1a9826769b54db089ba41c6e9fde0f5"
format_stamp: "Formatted at 2026-06-19 09:01:22 on dist-test-slave-mtdr"
server_key: "f58b6b669807656dc678edf00cc800fc"
server_key_iv: "214b5d756e480565ec06a21fa0806246"
server_key_version: "encryptionkey@0"
I20260619 09:01:22.106093 4701 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781859658134510-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781859658134510-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781859658134510-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 09:01:22.119472 4701 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 09:01:22.121217 4701 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 09:01:22.121356 4701 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 09:01:22.121562 4701 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 09:01:22.121881 4701 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 09:01:22.121917 4701 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 09:01:22.121948 4701 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 09:01:22.121976 4701 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 09:01:22.127662 4701 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38607
I20260619 09:01:22.127719 4828 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38607 every 8 connection(s)
I20260619 09:01:22.127945 4701 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781859658134510-392-0/minicluster-data/ts-1/data/info.pb
Jun 19 09:01:22 dist-test-slave-mtdr krb5kdc[4462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859682, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 09:01:22.136539 4546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:22.128576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33253 (local address 127.0.98.62:33019)
0619 09:01:22.128655 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:22.128657 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:22.128794 (+ 137us) server_negotiation.cc:407] Connection header received
0619 09:01:22.129251 (+ 457us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:22.129252 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:22.129261 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:22.129309 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:01:22.130186 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:22.130683 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:22.131672 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:22.131862 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:22.131910 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:22.134046 (+ 2136us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:01:22.134047 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:01:22.134047 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:01:22.134054 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:01:22.135422 (+ 1368us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:01:22.135801 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:01:22.135801 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:01:22.135802 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:01:22.135845 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:01:22.136024 (+ 179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:01:22.136025 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:01:22.136025 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:01:22.136112 (+ 87us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:01:22.136156 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0619 09:01:22.136417 (+ 261us) server_negotiation.cc:299] Negotiation successful
0619 09:01:22.136477 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 09:01:22.136730 4831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:22.128539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33019 (local address 127.0.98.2:33253)
0619 09:01:22.128680 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0619 09:01:22.128716 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0619 09:01:22.129170 (+ 454us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:01:22.129365 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:01:22.129368 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:01:22.129604 (+ 236us) client_negotiation.cc:189] Negotiated authn=SASL
0619 09:01:22.130086 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:22.130091 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:22.130769 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:22.130771 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:22.131598 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:22.131600 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:22.131948 (+ 348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:22.131949 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:22.132060 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:22.132629 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 09:01:22.132640 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 09:01:22.133959 (+ 1319us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 09:01:22.135504 (+ 1545us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:01:22.135510 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:01:22.135512 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:01:22.135736 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:01:22.135890 (+ 154us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:01:22.135891 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:01:22.135891 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:01:22.135979 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:01:22.136182 (+ 203us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 09:01:22.136187 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 09:01:22.136309 (+ 122us) client_negotiation.cc:769] Sending connection context
0619 09:01:22.136397 (+ 88us) client_negotiation.cc:240] Negotiation successful
0619 09:01:22.136513 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":81,"threads_started":1}
I20260619 09:01:22.137131 4829 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33019
I20260619 09:01:22.137213 4829 heartbeater.cc:464] Registering TS with master...
I20260619 09:01:22.137346 4829 heartbeater.cc:511] Master 127.0.98.62:33019 requested a full tablet report, sending...
I20260619 09:01:22.137634 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ez5NB/build/release/bin/kudu as pid 4701
I20260619 09:01:22.137692 4506 ts_manager.cc:194] Registered new tserver with Master: f1a9826769b54db089ba41c6e9fde0f5 (127.0.98.2:38607)
I20260619 09:01:22.137728 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781859658134510-392-0/minicluster-data/ts-1/wal/instance
I20260619 09:01:22.137887 392 external_mini_cluster.cc:1428] Setting key dfa1414cb22d4f47ec52c7da26e22ad6
I20260619 09:01:22.138111 4506 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:33253
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781859658134510-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781859658134510-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 09:01:22.161084 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ez5NB/build/release/bin/kudu
/tmp/dist-test-task4ez5NB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781859658134510-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781859658134510-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781859658134510-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781859658134510-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33019
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781859658134510-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781859658134510-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781859658134510-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781859658134510-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 09:01:22.237886 4834 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 09:01:22.238076 4834 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 09:01:22.238099 4834 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 09:01:22.239188 4834 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 09:01:22.239220 4834 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 09:01:22.239245 4834 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 09:01:22.240625 4834 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781859658134510-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781859658134510-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781859658134510-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781859658134510-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33019
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4834
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781859658134510-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 09:01:22.240799 4834 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 09:01:22.240957 4834 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 09:01:22.245528 4840 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:01:22.245536 4839 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:01:22.245579 4842 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:01:22.245673 4834 server_base.cc:1034] running on GCE node
I20260619 09:01:22.245929 4834 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 09:01:22.246145 4834 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 09:01:22.247289 4834 hybrid_clock.cc:648] HybridClock initialized: now 1781859682247268 us; error 34 us; skew 500 ppm
Jun 19 09:01:22 dist-test-slave-mtdr krb5kdc[4462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859682, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 09:01:22.249747 4834 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 09:01:22.250041 4834 fs_manager.cc:359] Metadata directory not provided
I20260619 09:01:22.250085 4834 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 09:01:22.250174 4834 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 09:01:22.251180 4834 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781859658134510-392-0/minicluster-data/ts-2/data/instance:
uuid: "298d9a1dea0f44f998ac0f58351d6f4b"
format_stamp: "Formatted at 2026-06-19 09:01:22 on dist-test-slave-mtdr"
server_key: "e4c7023490412621ce34039ba0276e36"
server_key_iv: "3b874327bf59a68a752eeff912164860"
server_key_version: "encryptionkey@0"
I20260619 09:01:22.251458 4834 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781859658134510-392-0/minicluster-data/ts-2/wal/instance:
uuid: "298d9a1dea0f44f998ac0f58351d6f4b"
format_stamp: "Formatted at 2026-06-19 09:01:22 on dist-test-slave-mtdr"
server_key: "e4c7023490412621ce34039ba0276e36"
server_key_iv: "3b874327bf59a68a752eeff912164860"
server_key_version: "encryptionkey@0"
I20260619 09:01:22.252568 4834 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 09:01:22.253249 4848 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 09:01:22.253410 4834 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 09:01:22.253500 4834 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781859658134510-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781859658134510-392-0/minicluster-data/ts-2/wal
uuid: "298d9a1dea0f44f998ac0f58351d6f4b"
format_stamp: "Formatted at 2026-06-19 09:01:22 on dist-test-slave-mtdr"
server_key: "e4c7023490412621ce34039ba0276e36"
server_key_iv: "3b874327bf59a68a752eeff912164860"
server_key_version: "encryptionkey@0"
I20260619 09:01:22.253551 4834 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781859658134510-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781859658134510-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781859658134510-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 09:01:22.270411 4834 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 09:01:22.272472 4834 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 09:01:22.272576 4834 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 09:01:22.272743 4834 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 09:01:22.272995 4834 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 09:01:22.273031 4834 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 09:01:22.273051 4834 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 09:01:22.273062 4834 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 09:01:22.278357 4834 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43399
I20260619 09:01:22.278393 4961 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43399 every 8 connection(s)
I20260619 09:01:22.278641 4834 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781859658134510-392-0/minicluster-data/ts-2/data/info.pb
Jun 19 09:01:22 dist-test-slave-mtdr krb5kdc[4462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859682, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 09:01:22.285269 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ez5NB/build/release/bin/kudu as pid 4834
I20260619 09:01:22.285364 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781859658134510-392-0/minicluster-data/ts-2/wal/instance
I20260619 09:01:22.285573 392 external_mini_cluster.cc:1428] Setting key ceed281eba6b0c0be41e29b18a0d441c
I20260619 09:01:22.287225 4546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:22.279298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59847 (local address 127.0.98.62:33019)
0619 09:01:22.279350 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:22.279351 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:22.279558 (+ 207us) server_negotiation.cc:407] Connection header received
0619 09:01:22.280008 (+ 450us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:22.280010 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:22.280019 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:22.280068 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:01:22.280886 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:22.281376 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:22.282349 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:22.282518 (+ 169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:22.282561 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:22.284597 (+ 2036us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:01:22.284599 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:01:22.284599 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:01:22.284605 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:01:22.285956 (+ 1351us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:01:22.286347 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:01:22.286349 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:01:22.286349 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:01:22.286404 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:01:22.286614 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:01:22.286614 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:01:22.286615 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:01:22.286729 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:01:22.286782 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0619 09:01:22.287103 (+ 321us) server_negotiation.cc:299] Negotiation successful
0619 09:01:22.287164 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260619 09:01:22.287389 4964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:22.279262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33019 (local address 127.0.98.3:59847)
0619 09:01:22.279442 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0619 09:01:22.279482 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 09:01:22.279922 (+ 440us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:01:22.280121 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:01:22.280124 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:01:22.280360 (+ 236us) client_negotiation.cc:189] Negotiated authn=SASL
0619 09:01:22.280813 (+ 453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:22.280818 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:22.281467 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:22.281468 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:22.282282 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:22.282285 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:22.282572 (+ 287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:22.282573 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:22.282668 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:22.283155 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 09:01:22.283164 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0619 09:01:22.284497 (+ 1333us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 09:01:22.286023 (+ 1526us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:01:22.286030 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:01:22.286032 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:01:22.286276 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:01:22.286458 (+ 182us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:01:22.286458 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:01:22.286459 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:01:22.286561 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:01:22.286808 (+ 247us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 09:01:22.286836 (+ 28us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 09:01:22.286990 (+ 154us) client_negotiation.cc:769] Sending connection context
0619 09:01:22.287081 (+ 91us) client_negotiation.cc:240] Negotiation successful
0619 09:01:22.287200 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":107,"threads_started":1}
I20260619 09:01:22.287776 4962 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33019
I20260619 09:01:22.287863 4962 heartbeater.cc:464] Registering TS with master...
I20260619 09:01:22.288014 4962 heartbeater.cc:511] Master 127.0.98.62:33019 requested a full tablet report, sending...
I20260619 09:01:22.288389 4505 ts_manager.cc:194] Registered new tserver with Master: 298d9a1dea0f44f998ac0f58351d6f4b (127.0.98.3:43399)
I20260619 09:01:22.288717 4505 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:59847
I20260619 09:01:22.297328 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 09:01:22.304123 4546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:22.298516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40956 (local address 127.0.98.62:33019)
0619 09:01:22.298571 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:22.298574 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:22.298673 (+ 99us) server_negotiation.cc:407] Connection header received
0619 09:01:22.298737 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:22.298738 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:22.298747 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:22.298788 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:01:22.299346 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:22.299805 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:22.300511 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:22.300692 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:22.300743 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:22.301827 (+ 1084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:01:22.301828 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:01:22.301828 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:01:22.301835 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:01:22.303095 (+ 1260us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:01:22.303395 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:01:22.303396 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:01:22.303404 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:01:22.303461 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:01:22.303618 (+ 157us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:01:22.303619 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:01:22.303619 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:01:22.303753 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:01:22.303837 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0619 09:01:22.303966 (+ 129us) server_negotiation.cc:299] Negotiation successful
0619 09:01:22.304035 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260619 09:01:22.305490 4505 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40956:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 09:01:22.306979 4505 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 09:01:22.313257 4970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:22.308969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43399 (local address 127.0.0.1:60330)
0619 09:01:22.309099 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0619 09:01:22.309136 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0619 09:01:22.309198 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:01:22.309697 (+ 499us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:01:22.309697 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:01:22.309699 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 09:01:22.309751 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:22.309753 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:22.310739 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:22.310740 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:22.311980 (+ 1240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:22.311983 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:22.313014 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:22.313015 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:22.313088 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:22.313100 (+ 12us) client_negotiation.cc:769] Sending connection context
0619 09:01:22.313150 (+ 50us) client_negotiation.cc:240] Negotiation successful
0619 09:01:22.313189 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":80,"threads_started":1}
I20260619 09:01:22.313283 4975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:22.309207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60330 (local address 127.0.98.3:43399)
0619 09:01:22.309550 (+ 343us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:22.309554 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:22.309562 (+ 8us) server_negotiation.cc:407] Connection header received
0619 09:01:22.309593 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:22.309596 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:22.309609 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:22.309819 (+ 210us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 09:01:22.309918 (+ 99us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:22.310614 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:22.312048 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:22.312933 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:22.313009 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:22.313066 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0619 09:01:22.313163 (+ 97us) server_negotiation.cc:299] Negotiation successful
0619 09:01:22.313202 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":316,"thread_start_us":117,"threads_started":1}
I20260619 09:01:22.313511 4974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:22.309212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51404 (local address 127.0.98.2:38607)
0619 09:01:22.309440 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:22.309444 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:22.309456 (+ 12us) server_negotiation.cc:407] Connection header received
0619 09:01:22.309518 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:22.309520 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:22.309534 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:22.309605 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 09:01:22.309923 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:22.311008 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:22.312235 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:22.312960 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:22.313015 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:22.313066 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0619 09:01:22.313251 (+ 185us) server_negotiation.cc:299] Negotiation successful
0619 09:01:22.313303 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":202,"thread_start_us":128,"threads_started":1}
I20260619 09:01:22.313741 4972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:22.308974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38607 (local address 127.0.0.1:51404)
0619 09:01:22.309395 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0619 09:01:22.309403 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0619 09:01:22.309447 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:01:22.309663 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:01:22.309666 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:01:22.309668 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 09:01:22.309732 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:22.309734 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:22.311112 (+ 1378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:22.311113 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:22.312165 (+ 1052us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:22.312167 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:22.313101 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:22.313102 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:22.313164 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:22.313169 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 09:01:22.313220 (+ 51us) client_negotiation.cc:240] Negotiation successful
0619 09:01:22.313259 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":409,"mutex_wait_us":4,"thread_start_us":69,"threads_started":1}
I20260619 09:01:22.314005 4896 tablet_service.cc:1467] Processing CreateTablet for tablet 3ec9f4458fcf4535881ba520544b30b1 (DEFAULT_TABLE table=test-table [id=6d0b9946aa044928b66af3a32f0a9114]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 09:01:22.314343 4973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:22.309212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51886 (local address 127.0.98.1:38021)
0619 09:01:22.309373 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:22.309378 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:22.309450 (+ 72us) server_negotiation.cc:407] Connection header received
0619 09:01:22.309536 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:22.309538 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:22.309553 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:22.309644 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 09:01:22.309840 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:22.311010 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:22.313367 (+ 2357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:22.314005 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:22.314066 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:22.314118 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0619 09:01:22.314230 (+ 112us) server_negotiation.cc:299] Negotiation successful
0619 09:01:22.314270 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":85,"threads_started":1}
I20260619 09:01:22.314360 4896 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ec9f4458fcf4535881ba520544b30b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 09:01:22.314788 4763 tablet_service.cc:1467] Processing CreateTablet for tablet 3ec9f4458fcf4535881ba520544b30b1 (DEFAULT_TABLE table=test-table [id=6d0b9946aa044928b66af3a32f0a9114]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 09:01:22.315086 4763 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ec9f4458fcf4535881ba520544b30b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 09:01:22.316901 4976 tablet_bootstrap.cc:492] T 3ec9f4458fcf4535881ba520544b30b1 P 298d9a1dea0f44f998ac0f58351d6f4b: Bootstrap starting.
I20260619 09:01:22.317423 4976 tablet_bootstrap.cc:654] T 3ec9f4458fcf4535881ba520544b30b1 P 298d9a1dea0f44f998ac0f58351d6f4b: Neither blocks nor log segments found. Creating new log.
I20260619 09:01:22.317444 4977 tablet_bootstrap.cc:492] T 3ec9f4458fcf4535881ba520544b30b1 P f1a9826769b54db089ba41c6e9fde0f5: Bootstrap starting.
I20260619 09:01:22.317577 4971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:22.308974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38021 (local address 127.0.0.1:51886)
0619 09:01:22.309387 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0619 09:01:22.309399 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 09:01:22.309462 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:01:22.309726 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:01:22.309728 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:01:22.309729 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 09:01:22.309773 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:22.309774 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:22.312228 (+ 2454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:22.312228 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:22.312941 (+ 713us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:22.312943 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:22.314076 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:22.314077 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:22.314136 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:22.314139 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 09:01:22.314192 (+ 53us) client_negotiation.cc:240] Negotiation successful
0619 09:01:22.314229 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":68,"threads_started":1}
I20260619 09:01:22.317924 4976 log.cc:824] T 3ec9f4458fcf4535881ba520544b30b1 P 298d9a1dea0f44f998ac0f58351d6f4b: Log is configured to *not* fsync() on all Append() calls
I20260619 09:01:22.318046 4977 tablet_bootstrap.cc:654] T 3ec9f4458fcf4535881ba520544b30b1 P f1a9826769b54db089ba41c6e9fde0f5: Neither blocks nor log segments found. Creating new log.
I20260619 09:01:22.318220 4630 tablet_service.cc:1467] Processing CreateTablet for tablet 3ec9f4458fcf4535881ba520544b30b1 (DEFAULT_TABLE table=test-table [id=6d0b9946aa044928b66af3a32f0a9114]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 09:01:22.318547 4630 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ec9f4458fcf4535881ba520544b30b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 09:01:22.318572 4976 tablet_bootstrap.cc:492] T 3ec9f4458fcf4535881ba520544b30b1 P 298d9a1dea0f44f998ac0f58351d6f4b: No bootstrap required, opened a new log
I20260619 09:01:22.318572 4977 log.cc:824] T 3ec9f4458fcf4535881ba520544b30b1 P f1a9826769b54db089ba41c6e9fde0f5: Log is configured to *not* fsync() on all Append() calls
I20260619 09:01:22.318625 4976 ts_tablet_manager.cc:1397] T 3ec9f4458fcf4535881ba520544b30b1 P 298d9a1dea0f44f998ac0f58351d6f4b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 09:01:22.319276 4977 tablet_bootstrap.cc:492] T 3ec9f4458fcf4535881ba520544b30b1 P f1a9826769b54db089ba41c6e9fde0f5: No bootstrap required, opened a new log
I20260619 09:01:22.319341 4977 ts_tablet_manager.cc:1397] T 3ec9f4458fcf4535881ba520544b30b1 P f1a9826769b54db089ba41c6e9fde0f5: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 09:01:22.320061 4976 raft_consensus.cc:348] T 3ec9f4458fcf4535881ba520544b30b1 P 298d9a1dea0f44f998ac0f58351d6f4b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c1f48243a8c4435915b694698b999c5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38021 } } peers { permanent_uuid: "298d9a1dea0f44f998ac0f58351d6f4b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43399 } } peers { permanent_uuid: "f1a9826769b54db089ba41c6e9fde0f5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38607 } }
I20260619 09:01:22.320215 4976 raft_consensus.cc:374] T 3ec9f4458fcf4535881ba520544b30b1 P 298d9a1dea0f44f998ac0f58351d6f4b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 09:01:22.320250 4976 raft_consensus.cc:729] T 3ec9f4458fcf4535881ba520544b30b1 P 298d9a1dea0f44f998ac0f58351d6f4b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 298d9a1dea0f44f998ac0f58351d6f4b, State: Initialized, Role: FOLLOWER
I20260619 09:01:22.320356 4976 consensus_queue.cc:260] T 3ec9f4458fcf4535881ba520544b30b1 P 298d9a1dea0f44f998ac0f58351d6f4b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c1f48243a8c4435915b694698b999c5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38021 } } peers { permanent_uuid: "298d9a1dea0f44f998ac0f58351d6f4b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43399 } } peers { permanent_uuid: "f1a9826769b54db089ba41c6e9fde0f5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38607 } }
I20260619 09:01:22.320605 4976 ts_tablet_manager.cc:1428] T 3ec9f4458fcf4535881ba520544b30b1 P 298d9a1dea0f44f998ac0f58351d6f4b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 09:01:22.320731 4962 heartbeater.cc:503] Master 127.0.98.62:33019 was elected leader, sending a full tablet report...
I20260619 09:01:22.321194 4977 raft_consensus.cc:348] T 3ec9f4458fcf4535881ba520544b30b1 P f1a9826769b54db089ba41c6e9fde0f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c1f48243a8c4435915b694698b999c5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38021 } } peers { permanent_uuid: "298d9a1dea0f44f998ac0f58351d6f4b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43399 } } peers { permanent_uuid: "f1a9826769b54db089ba41c6e9fde0f5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38607 } }
I20260619 09:01:22.321339 4977 raft_consensus.cc:374] T 3ec9f4458fcf4535881ba520544b30b1 P f1a9826769b54db089ba41c6e9fde0f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 09:01:22.321375 4977 raft_consensus.cc:729] T 3ec9f4458fcf4535881ba520544b30b1 P f1a9826769b54db089ba41c6e9fde0f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1a9826769b54db089ba41c6e9fde0f5, State: Initialized, Role: FOLLOWER
I20260619 09:01:22.321570 4981 tablet_bootstrap.cc:492] T 3ec9f4458fcf4535881ba520544b30b1 P 3c1f48243a8c4435915b694698b999c5: Bootstrap starting.
I20260619 09:01:22.321516 4977 consensus_queue.cc:260] T 3ec9f4458fcf4535881ba520544b30b1 P f1a9826769b54db089ba41c6e9fde0f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c1f48243a8c4435915b694698b999c5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38021 } } peers { permanent_uuid: "298d9a1dea0f44f998ac0f58351d6f4b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43399 } } peers { permanent_uuid: "f1a9826769b54db089ba41c6e9fde0f5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38607 } }
I20260619 09:01:22.321841 4829 heartbeater.cc:503] Master 127.0.98.62:33019 was elected leader, sending a full tablet report...
I20260619 09:01:22.321841 4977 ts_tablet_manager.cc:1428] T 3ec9f4458fcf4535881ba520544b30b1 P f1a9826769b54db089ba41c6e9fde0f5: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260619 09:01:22.322448 4981 tablet_bootstrap.cc:654] T 3ec9f4458fcf4535881ba520544b30b1 P 3c1f48243a8c4435915b694698b999c5: Neither blocks nor log segments found. Creating new log.
I20260619 09:01:22.323081 4981 log.cc:824] T 3ec9f4458fcf4535881ba520544b30b1 P 3c1f48243a8c4435915b694698b999c5: Log is configured to *not* fsync() on all Append() calls
I20260619 09:01:22.323732 4981 tablet_bootstrap.cc:492] T 3ec9f4458fcf4535881ba520544b30b1 P 3c1f48243a8c4435915b694698b999c5: No bootstrap required, opened a new log
I20260619 09:01:22.323791 4981 ts_tablet_manager.cc:1397] T 3ec9f4458fcf4535881ba520544b30b1 P 3c1f48243a8c4435915b694698b999c5: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260619 09:01:22.325305 4981 raft_consensus.cc:348] T 3ec9f4458fcf4535881ba520544b30b1 P 3c1f48243a8c4435915b694698b999c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c1f48243a8c4435915b694698b999c5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38021 } } peers { permanent_uuid: "298d9a1dea0f44f998ac0f58351d6f4b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43399 } } peers { permanent_uuid: "f1a9826769b54db089ba41c6e9fde0f5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38607 } }
I20260619 09:01:22.325430 4981 raft_consensus.cc:374] T 3ec9f4458fcf4535881ba520544b30b1 P 3c1f48243a8c4435915b694698b999c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 09:01:22.325453 4981 raft_consensus.cc:729] T 3ec9f4458fcf4535881ba520544b30b1 P 3c1f48243a8c4435915b694698b999c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c1f48243a8c4435915b694698b999c5, State: Initialized, Role: FOLLOWER
I20260619 09:01:22.325529 4981 consensus_queue.cc:260] T 3ec9f4458fcf4535881ba520544b30b1 P 3c1f48243a8c4435915b694698b999c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c1f48243a8c4435915b694698b999c5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38021 } } peers { permanent_uuid: "298d9a1dea0f44f998ac0f58351d6f4b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43399 } } peers { permanent_uuid: "f1a9826769b54db089ba41c6e9fde0f5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38607 } }
I20260619 09:01:22.325731 4981 ts_tablet_manager.cc:1428] T 3ec9f4458fcf4535881ba520544b30b1 P 3c1f48243a8c4435915b694698b999c5: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 09:01:22.325758 4696 heartbeater.cc:503] Master 127.0.98.62:33019 was elected leader, sending a full tablet report...
W20260619 09:01:22.378402 4830 tablet.cc:2369] T 3ec9f4458fcf4535881ba520544b30b1 P f1a9826769b54db089ba41c6e9fde0f5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 09:01:22.449074 4984 raft_consensus.cc:482] T 3ec9f4458fcf4535881ba520544b30b1 P 3c1f48243a8c4435915b694698b999c5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 09:01:22.449196 4984 raft_consensus.cc:504] T 3ec9f4458fcf4535881ba520544b30b1 P 3c1f48243a8c4435915b694698b999c5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c1f48243a8c4435915b694698b999c5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38021 } } peers { permanent_uuid: "298d9a1dea0f44f998ac0f58351d6f4b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43399 } } peers { permanent_uuid: "f1a9826769b54db089ba41c6e9fde0f5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38607 } }
I20260619 09:01:22.449517 4984 leader_election.cc:288] T 3ec9f4458fcf4535881ba520544b30b1 P 3c1f48243a8c4435915b694698b999c5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 298d9a1dea0f44f998ac0f58351d6f4b (127.0.98.3:43399), f1a9826769b54db089ba41c6e9fde0f5 (127.0.98.2:38607)
I20260619 09:01:22.452265 4698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:22.449690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43399 (local address 127.0.98.1:44393)
0619 09:01:22.449733 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0619 09:01:22.449746 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 09:01:22.449792 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:01:22.449927 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:01:22.449928 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:01:22.449930 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 09:01:22.450008 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:22.450010 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:22.450657 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:22.450658 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:22.451454 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:22.451456 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:22.452032 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:22.452032 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:22.452106 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:22.452109 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 09:01:22.452153 (+ 44us) client_negotiation.cc:240] Negotiation successful
0619 09:01:22.452190 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22}
I20260619 09:01:22.452396 4975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:22.449753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44393 (local address 127.0.98.3:43399)
0619 09:01:22.449806 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:22.449808 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:22.449818 (+ 10us) server_negotiation.cc:407] Connection header received
0619 09:01:22.449845 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:22.449846 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:22.449856 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:22.449908 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 09:01:22.450067 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:22.450572 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:22.451536 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:22.451930 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:22.451992 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:22.452002 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0619 09:01:22.452234 (+ 232us) server_negotiation.cc:299] Negotiation successful
0619 09:01:22.452325 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260619 09:01:22.452682 4985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:22.449696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38607 (local address 127.0.98.1:41391)
0619 09:01:22.450020 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0619 09:01:22.450029 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 09:01:22.450088 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:01:22.450235 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:01:22.450236 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:01:22.450238 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 09:01:22.450307 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:22.450309 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:22.450990 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:22.450991 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:22.451730 (+ 739us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:22.451732 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:22.452446 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:22.452448 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:22.452526 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:22.452530 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 09:01:22.452577 (+ 47us) client_negotiation.cc:240] Negotiation successful
0619 09:01:22.452610 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":307,"mutex_wait_us":152,"thread_start_us":99,"threads_started":1}
I20260619 09:01:22.452683 4974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:22.449755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41391 (local address 127.0.98.2:38607)
0619 09:01:22.449806 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:22.449808 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:22.450095 (+ 287us) server_negotiation.cc:407] Connection header received
0619 09:01:22.450162 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:22.450163 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:22.450177 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:22.450228 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 09:01:22.450410 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:22.450920 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:22.451815 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:22.452342 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:22.452415 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:22.452425 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0619 09:01:22.452584 (+ 159us) server_negotiation.cc:299] Negotiation successful
0619 09:01:22.452624 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260619 09:01:22.452687 4916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ec9f4458fcf4535881ba520544b30b1" candidate_uuid: "3c1f48243a8c4435915b694698b999c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "298d9a1dea0f44f998ac0f58351d6f4b" is_pre_election: true
I20260619 09:01:22.452885 4916 raft_consensus.cc:2454] T 3ec9f4458fcf4535881ba520544b30b1 P 298d9a1dea0f44f998ac0f58351d6f4b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c1f48243a8c4435915b694698b999c5 in term 0.
I20260619 09:01:22.453138 4586 leader_election.cc:302] T 3ec9f4458fcf4535881ba520544b30b1 P 3c1f48243a8c4435915b694698b999c5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 298d9a1dea0f44f998ac0f58351d6f4b, 3c1f48243a8c4435915b694698b999c5; no voters:
I20260619 09:01:22.453258 4783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ec9f4458fcf4535881ba520544b30b1" candidate_uuid: "3c1f48243a8c4435915b694698b999c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1a9826769b54db089ba41c6e9fde0f5" is_pre_election: true
I20260619 09:01:22.453313 4984 raft_consensus.cc:2781] T 3ec9f4458fcf4535881ba520544b30b1 P 3c1f48243a8c4435915b694698b999c5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 09:01:22.453384 4783 raft_consensus.cc:2454] T 3ec9f4458fcf4535881ba520544b30b1 P f1a9826769b54db089ba41c6e9fde0f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c1f48243a8c4435915b694698b999c5 in term 0.
I20260619 09:01:22.453403 4984 raft_consensus.cc:482] T 3ec9f4458fcf4535881ba520544b30b1 P 3c1f48243a8c4435915b694698b999c5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 09:01:22.453434 4984 raft_consensus.cc:3037] T 3ec9f4458fcf4535881ba520544b30b1 P 3c1f48243a8c4435915b694698b999c5 [term 0 FOLLOWER]: Advancing to term 1
I20260619 09:01:22.454113 4984 raft_consensus.cc:504] T 3ec9f4458fcf4535881ba520544b30b1 P 3c1f48243a8c4435915b694698b999c5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c1f48243a8c4435915b694698b999c5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38021 } } peers { permanent_uuid: "298d9a1dea0f44f998ac0f58351d6f4b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43399 } } peers { permanent_uuid: "f1a9826769b54db089ba41c6e9fde0f5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38607 } }
I20260619 09:01:22.454248 4984 leader_election.cc:288] T 3ec9f4458fcf4535881ba520544b30b1 P 3c1f48243a8c4435915b694698b999c5 [CANDIDATE]: Term 1 election: Requested vote from peers 298d9a1dea0f44f998ac0f58351d6f4b (127.0.98.3:43399), f1a9826769b54db089ba41c6e9fde0f5 (127.0.98.2:38607)
I20260619 09:01:22.454383 4916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ec9f4458fcf4535881ba520544b30b1" candidate_uuid: "3c1f48243a8c4435915b694698b999c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "298d9a1dea0f44f998ac0f58351d6f4b"
I20260619 09:01:22.454419 4783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ec9f4458fcf4535881ba520544b30b1" candidate_uuid: "3c1f48243a8c4435915b694698b999c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1a9826769b54db089ba41c6e9fde0f5"
I20260619 09:01:22.454465 4916 raft_consensus.cc:3037] T 3ec9f4458fcf4535881ba520544b30b1 P 298d9a1dea0f44f998ac0f58351d6f4b [term 0 FOLLOWER]: Advancing to term 1
I20260619 09:01:22.454483 4783 raft_consensus.cc:3037] T 3ec9f4458fcf4535881ba520544b30b1 P f1a9826769b54db089ba41c6e9fde0f5 [term 0 FOLLOWER]: Advancing to term 1
I20260619 09:01:22.455201 4783 raft_consensus.cc:2454] T 3ec9f4458fcf4535881ba520544b30b1 P f1a9826769b54db089ba41c6e9fde0f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c1f48243a8c4435915b694698b999c5 in term 1.
I20260619 09:01:22.455206 4916 raft_consensus.cc:2454] T 3ec9f4458fcf4535881ba520544b30b1 P 298d9a1dea0f44f998ac0f58351d6f4b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c1f48243a8c4435915b694698b999c5 in term 1.
I20260619 09:01:22.455372 4585 leader_election.cc:302] T 3ec9f4458fcf4535881ba520544b30b1 P 3c1f48243a8c4435915b694698b999c5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c1f48243a8c4435915b694698b999c5, f1a9826769b54db089ba41c6e9fde0f5; no voters:
I20260619 09:01:22.455469 4984 raft_consensus.cc:2781] T 3ec9f4458fcf4535881ba520544b30b1 P 3c1f48243a8c4435915b694698b999c5 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 09:01:22.455655 4984 raft_consensus.cc:686] T 3ec9f4458fcf4535881ba520544b30b1 P 3c1f48243a8c4435915b694698b999c5 [term 1 LEADER]: Becoming Leader. State: Replica: 3c1f48243a8c4435915b694698b999c5, State: Running, Role: LEADER
I20260619 09:01:22.455739 4984 consensus_queue.cc:237] T 3ec9f4458fcf4535881ba520544b30b1 P 3c1f48243a8c4435915b694698b999c5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c1f48243a8c4435915b694698b999c5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38021 } } peers { permanent_uuid: "298d9a1dea0f44f998ac0f58351d6f4b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43399 } } peers { permanent_uuid: "f1a9826769b54db089ba41c6e9fde0f5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38607 } }
I20260619 09:01:22.456635 4505 catalog_manager.cc:5495] T 3ec9f4458fcf4535881ba520544b30b1 P 3c1f48243a8c4435915b694698b999c5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3c1f48243a8c4435915b694698b999c5 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "3c1f48243a8c4435915b694698b999c5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c1f48243a8c4435915b694698b999c5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "298d9a1dea0f44f998ac0f58351d6f4b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43399 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f1a9826769b54db089ba41c6e9fde0f5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38607 } health_report { overall_health: UNKNOWN } } }
W20260619 09:01:22.487187 4697 tablet.cc:2369] T 3ec9f4458fcf4535881ba520544b30b1 P 3c1f48243a8c4435915b694698b999c5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 09:01:22.491866 4973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:22.489785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51890 (local address 127.0.98.1:38021)
0619 09:01:22.489821 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:22.489823 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:22.489829 (+ 6us) server_negotiation.cc:407] Connection header received
0619 09:01:22.489870 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:22.489871 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:22.489887 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:22.489944 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 09:01:22.490051 (+ 107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:22.490496 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:22.491225 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:22.491400 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:22.491443 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:22.491563 (+ 120us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 09:01:22.491671 (+ 108us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 09:01:22.491729 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0619 09:01:22.491795 (+ 66us) server_negotiation.cc:299] Negotiation successful
0619 09:01:22.491813 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260619 09:01:22.493381 4783 raft_consensus.cc:1261] T 3ec9f4458fcf4535881ba520544b30b1 P f1a9826769b54db089ba41c6e9fde0f5 [term 1 FOLLOWER]: Refusing update from remote peer 3c1f48243a8c4435915b694698b999c5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 09:01:22.493382 4916 raft_consensus.cc:1261] T 3ec9f4458fcf4535881ba520544b30b1 P 298d9a1dea0f44f998ac0f58351d6f4b [term 1 FOLLOWER]: Refusing update from remote peer 3c1f48243a8c4435915b694698b999c5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 09:01:22.493716 4986 consensus_queue.cc:1035] T 3ec9f4458fcf4535881ba520544b30b1 P 3c1f48243a8c4435915b694698b999c5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "298d9a1dea0f44f998ac0f58351d6f4b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 09:01:22.493839 4984 consensus_queue.cc:1035] T 3ec9f4458fcf4535881ba520544b30b1 P 3c1f48243a8c4435915b694698b999c5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1a9826769b54db089ba41c6e9fde0f5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 09:01:22.495879 4991 mvcc.cc:205] Tried to move back new op lower bound from 7298497259490594816 to 7298497259338137600. Current Snapshot: MvccSnapshot[applied={T|T < 7298497259490594816}]
I20260619 09:01:22.498248 4506 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40956:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 09:01:22.498322 4506 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40956:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 09:01:22.498972 4506 catalog_manager.cc:5782] T 00000000000000000000000000000000 P aa813b46e3904b58bac3fd579acd3e81: Sending DeleteTablet for 3 replicas of tablet 3ec9f4458fcf4535881ba520544b30b1
I20260619 09:01:22.499266 4896 tablet_service.cc:1514] Processing DeleteTablet for tablet 3ec9f4458fcf4535881ba520544b30b1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 09:01:22 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60330
I20260619 09:01:22.499303 4630 tablet_service.cc:1514] Processing DeleteTablet for tablet 3ec9f4458fcf4535881ba520544b30b1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 09:01:22 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51886
I20260619 09:01:22.499497 4999 tablet_replica.cc:330] T 3ec9f4458fcf4535881ba520544b30b1 P 298d9a1dea0f44f998ac0f58351d6f4b: stopping tablet replica
I20260619 09:01:22.499473 4763 tablet_service.cc:1514] Processing DeleteTablet for tablet 3ec9f4458fcf4535881ba520544b30b1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 09:01:22 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51404
I20260619 09:01:22.499600 4999 raft_consensus.cc:2229] T 3ec9f4458fcf4535881ba520544b30b1 P 298d9a1dea0f44f998ac0f58351d6f4b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 09:01:22.499701 4999 raft_consensus.cc:2258] T 3ec9f4458fcf4535881ba520544b30b1 P 298d9a1dea0f44f998ac0f58351d6f4b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 09:01:22.499698 5001 tablet_replica.cc:330] T 3ec9f4458fcf4535881ba520544b30b1 P f1a9826769b54db089ba41c6e9fde0f5: stopping tablet replica
I20260619 09:01:22.499923 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ez5NB/build/release/bin/kudu with pid 4568
I20260619 09:01:22.500126 5001 raft_consensus.cc:2229] T 3ec9f4458fcf4535881ba520544b30b1 P f1a9826769b54db089ba41c6e9fde0f5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 09:01:22.500166 4999 ts_tablet_manager.cc:1905] T 3ec9f4458fcf4535881ba520544b30b1 P 298d9a1dea0f44f998ac0f58351d6f4b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 09:01:22.500509 5001 raft_consensus.cc:2258] T 3ec9f4458fcf4535881ba520544b30b1 P f1a9826769b54db089ba41c6e9fde0f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 09:01:22.500850 5001 ts_tablet_manager.cc:1905] T 3ec9f4458fcf4535881ba520544b30b1 P f1a9826769b54db089ba41c6e9fde0f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 09:01:22.501757 4999 ts_tablet_manager.cc:1918] T 3ec9f4458fcf4535881ba520544b30b1 P 298d9a1dea0f44f998ac0f58351d6f4b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 09:01:22.501824 4999 log.cc:1196] T 3ec9f4458fcf4535881ba520544b30b1 P 298d9a1dea0f44f998ac0f58351d6f4b: Deleting WAL directory at /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781859658134510-392-0/minicluster-data/ts-2/wal/wals/3ec9f4458fcf4535881ba520544b30b1
I20260619 09:01:22.501989 5001 ts_tablet_manager.cc:1918] T 3ec9f4458fcf4535881ba520544b30b1 P f1a9826769b54db089ba41c6e9fde0f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 09:01:22.502049 5001 log.cc:1196] T 3ec9f4458fcf4535881ba520544b30b1 P f1a9826769b54db089ba41c6e9fde0f5: Deleting WAL directory at /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781859658134510-392-0/minicluster-data/ts-1/wal/wals/3ec9f4458fcf4535881ba520544b30b1
I20260619 09:01:22.502065 4999 ts_tablet_manager.cc:1939] T 3ec9f4458fcf4535881ba520544b30b1 P 298d9a1dea0f44f998ac0f58351d6f4b: Deleting consensus metadata
I20260619 09:01:22.502266 5001 ts_tablet_manager.cc:1939] T 3ec9f4458fcf4535881ba520544b30b1 P f1a9826769b54db089ba41c6e9fde0f5: Deleting consensus metadata
I20260619 09:01:22.502388 4493 catalog_manager.cc:4841] TS 298d9a1dea0f44f998ac0f58351d6f4b (127.0.98.3:43399): tablet 3ec9f4458fcf4535881ba520544b30b1 (table test-table [id=6d0b9946aa044928b66af3a32f0a9114]) successfully deleted
I20260619 09:01:22.502593 4492 catalog_manager.cc:4841] TS f1a9826769b54db089ba41c6e9fde0f5 (127.0.98.2:38607): tablet 3ec9f4458fcf4535881ba520544b30b1 (table test-table [id=6d0b9946aa044928b66af3a32f0a9114]) successfully deleted
W20260619 09:01:22.505414 4490 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:38021 (error 108)
I20260619 09:01:22.505563 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ez5NB/build/release/bin/kudu with pid 4701
I20260619 09:01:22.506377 4971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:22.506249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38021 (local address 127.0.0.1:51896)
0619 09:01:22.506304 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0619 09:01:22.506347 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38021: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260619 09:01:22.506497 4490 catalog_manager.cc:4584] TS 3c1f48243a8c4435915b694698b999c5 (127.0.98.1:38021): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3ec9f4458fcf4535881ba520544b30b1: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38021: connect: Connection refused (error 111)
I20260619 09:01:22.509928 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ez5NB/build/release/bin/kudu with pid 4834
I20260619 09:01:22.513541 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ez5NB/build/release/bin/kudu with pid 4475
2026-06-19T09:01:22Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (1898 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-06-19T09:01:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T09:01:22Z Disabled control of system clock
I20260619 09:01:22.537518 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ez5NB/build/release/bin/kudu
/tmp/dist-test-task4ez5NB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781859658134510-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781859658134510-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781859658134510-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781859658134510-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37253
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:37451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37253
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260619 09:01:22.613268 5006 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 09:01:22.613451 5006 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 09:01:22.613469 5006 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 09:01:22.614545 5006 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 09:01:22.614581 5006 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 09:01:22.614593 5006 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 09:01:22.614603 5006 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 09:01:22.614614 5006 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 09:01:22.615917 5006 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781859658134510-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781859658134510-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37253
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37253
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781859658134510-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5006
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781859658134510-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 09:01:22.616093 5006 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 09:01:22.616297 5006 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 09:01:22.620957 5011 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:01:22.620944 5014 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:01:22.620946 5012 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:01:22.621032 5006 server_base.cc:1034] running on GCE node
I20260619 09:01:22.621438 5006 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 09:01:22.621717 5006 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 09:01:22.622871 5006 hybrid_clock.cc:648] HybridClock initialized: now 1781859682622854 us; error 29 us; skew 500 ppm
I20260619 09:01:22.624058 5006 webserver.cc:460] Webserver started at http://127.0.98.62:32889/ using document root <none> and password file <none>
I20260619 09:01:22.624265 5006 fs_manager.cc:359] Metadata directory not provided
I20260619 09:01:22.624310 5006 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 09:01:22.624433 5006 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 09:01:22.625443 5006 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781859658134510-392-0/minicluster-data/master-0/data/instance:
uuid: "bc3a10589f9c476485a0394e81062306"
format_stamp: "Formatted at 2026-06-19 09:01:22 on dist-test-slave-mtdr"
server_key: "a2606946357b1cb4f4f1a5b4357c0acb"
server_key_iv: "69d680fe640fe00e5437338ad044c50b"
server_key_version: "encryptionkey@0"
I20260619 09:01:22.625770 5006 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781859658134510-392-0/minicluster-data/master-0/wal/instance:
uuid: "bc3a10589f9c476485a0394e81062306"
format_stamp: "Formatted at 2026-06-19 09:01:22 on dist-test-slave-mtdr"
server_key: "a2606946357b1cb4f4f1a5b4357c0acb"
server_key_iv: "69d680fe640fe00e5437338ad044c50b"
server_key_version: "encryptionkey@0"
I20260619 09:01:22.626962 5006 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 09:01:22.627628 5020 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 09:01:22.627775 5006 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 09:01:22.627853 5006 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781859658134510-392-0/minicluster-data/master-0/data,/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781859658134510-392-0/minicluster-data/master-0/wal
uuid: "bc3a10589f9c476485a0394e81062306"
format_stamp: "Formatted at 2026-06-19 09:01:22 on dist-test-slave-mtdr"
server_key: "a2606946357b1cb4f4f1a5b4357c0acb"
server_key_iv: "69d680fe640fe00e5437338ad044c50b"
server_key_version: "encryptionkey@0"
I20260619 09:01:22.627919 5006 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781859658134510-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781859658134510-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781859658134510-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 09:01:22.631094 5006 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 09:01:22.631333 5006 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 09:01:22.631448 5006 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 09:01:22.635079 5006 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37253
I20260619 09:01:22.635120 5072 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37253 every 8 connection(s)
I20260619 09:01:22.635407 5006 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781859658134510-392-0/minicluster-data/master-0/data/info.pb
I20260619 09:01:22.635979 5073 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 09:01:22.638633 5073 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc3a10589f9c476485a0394e81062306: Bootstrap starting.
I20260619 09:01:22.639196 5073 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bc3a10589f9c476485a0394e81062306: Neither blocks nor log segments found. Creating new log.
I20260619 09:01:22.639439 5073 log.cc:824] T 00000000000000000000000000000000 P bc3a10589f9c476485a0394e81062306: Log is configured to *not* fsync() on all Append() calls
I20260619 09:01:22.640026 5073 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc3a10589f9c476485a0394e81062306: No bootstrap required, opened a new log
I20260619 09:01:22.641188 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ez5NB/build/release/bin/kudu as pid 5006
I20260619 09:01:22.641283 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781859658134510-392-0/minicluster-data/master-0/wal/instance
I20260619 09:01:22.641453 392 external_mini_cluster.cc:1428] Setting key 884a436c1f51369ededb8f9e1f5620e1
I20260619 09:01:22.641974 5073 raft_consensus.cc:348] T 00000000000000000000000000000000 P bc3a10589f9c476485a0394e81062306 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc3a10589f9c476485a0394e81062306" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37253 } }
I20260619 09:01:22.642144 5073 raft_consensus.cc:374] T 00000000000000000000000000000000 P bc3a10589f9c476485a0394e81062306 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 09:01:22.642170 5073 raft_consensus.cc:729] T 00000000000000000000000000000000 P bc3a10589f9c476485a0394e81062306 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc3a10589f9c476485a0394e81062306, State: Initialized, Role: FOLLOWER
I20260619 09:01:22.642289 5073 consensus_queue.cc:260] T 00000000000000000000000000000000 P bc3a10589f9c476485a0394e81062306 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc3a10589f9c476485a0394e81062306" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37253 } }
I20260619 09:01:22.642344 5073 raft_consensus.cc:388] T 00000000000000000000000000000000 P bc3a10589f9c476485a0394e81062306 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 09:01:22.642366 5073 raft_consensus.cc:482] T 00000000000000000000000000000000 P bc3a10589f9c476485a0394e81062306 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 09:01:22.642408 5073 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bc3a10589f9c476485a0394e81062306 [term 0 FOLLOWER]: Advancing to term 1
I20260619 09:01:22.643071 5073 raft_consensus.cc:504] T 00000000000000000000000000000000 P bc3a10589f9c476485a0394e81062306 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc3a10589f9c476485a0394e81062306" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37253 } }
I20260619 09:01:22.643177 5073 leader_election.cc:302] T 00000000000000000000000000000000 P bc3a10589f9c476485a0394e81062306 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bc3a10589f9c476485a0394e81062306; no voters:
I20260619 09:01:22.643326 5073 leader_election.cc:288] T 00000000000000000000000000000000 P bc3a10589f9c476485a0394e81062306 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 09:01:22.643415 5078 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bc3a10589f9c476485a0394e81062306 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 09:01:22.643566 5073 sys_catalog.cc:564] T 00000000000000000000000000000000 P bc3a10589f9c476485a0394e81062306 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 09:01:22.643587 5078 raft_consensus.cc:686] T 00000000000000000000000000000000 P bc3a10589f9c476485a0394e81062306 [term 1 LEADER]: Becoming Leader. State: Replica: bc3a10589f9c476485a0394e81062306, State: Running, Role: LEADER
I20260619 09:01:22.643700 5078 consensus_queue.cc:237] T 00000000000000000000000000000000 P bc3a10589f9c476485a0394e81062306 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc3a10589f9c476485a0394e81062306" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37253 } }
I20260619 09:01:22.643954 5077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:22.641939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51750 (local address 127.0.98.62:37253)
0619 09:01:22.642149 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:22.642158 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:22.642205 (+ 47us) server_negotiation.cc:407] Connection header received
0619 09:01:22.642317 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:22.642324 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:22.642366 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:22.642499 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:01:22.643186 (+ 687us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:01:22.643190 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:01:22.643191 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 09:01:22.643195 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:01:22.643240 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 09:01:22.643254 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:01:22.643314 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0619 09:01:22.643377 (+ 63us) server_negotiation.cc:299] Negotiation successful
0619 09:01:22.643524 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":91,"threads_started":1}
I20260619 09:01:22.644021 5078 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc3a10589f9c476485a0394e81062306 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bc3a10589f9c476485a0394e81062306. Latest consensus state: current_term: 1 leader_uuid: "bc3a10589f9c476485a0394e81062306" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc3a10589f9c476485a0394e81062306" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37253 } } }
I20260619 09:01:22.644079 5078 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc3a10589f9c476485a0394e81062306 [sys.catalog]: This master's current role is: LEADER
I20260619 09:01:22.644536 5079 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc3a10589f9c476485a0394e81062306 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bc3a10589f9c476485a0394e81062306" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc3a10589f9c476485a0394e81062306" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37253 } } }
I20260619 09:01:22.644613 5079 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc3a10589f9c476485a0394e81062306 [sys.catalog]: This master's current role is: LEADER
I20260619 09:01:22.644948 5086 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 09:01:22.645529 5086 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 09:01:22.646811 5086 catalog_manager.cc:1347] Generated new cluster ID: 41503801f7cc400cb72253a84f87b003
I20260619 09:01:22.646921 5086 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 09:01:22.659446 5086 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 09:01:22.659577 5086 catalog_manager.cc:1504] Loading token signing keys...
I20260619 09:01:22.666420 5086 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bc3a10589f9c476485a0394e81062306: Generated new TSK 0
I20260619 09:01:22.666575 5086 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 09:01:22.669342 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ez5NB/build/release/bin/kudu
/tmp/dist-test-task4ez5NB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781859658134510-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781859658134510-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781859658134510-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781859658134510-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37253
--builtin_ntp_servers=127.0.98.20:37451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260619 09:01:22.745939 5097 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 09:01:22.746132 5097 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 09:01:22.746160 5097 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 09:01:22.747334 5097 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 09:01:22.747370 5097 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 09:01:22.747404 5097 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 09:01:22.748708 5097 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781859658134510-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781859658134510-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781859658134510-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37253
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5097
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781859658134510-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 09:01:22.748922 5097 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 09:01:22.749177 5097 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 09:01:22.753703 5103 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:01:22.753841 5097 server_base.cc:1034] running on GCE node
W20260619 09:01:22.753899 5105 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:01:22.753711 5102 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:01:22.754148 5097 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 09:01:22.754395 5097 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 09:01:22.755550 5097 hybrid_clock.cc:648] HybridClock initialized: now 1781859682755538 us; error 25 us; skew 500 ppm
I20260619 09:01:22.756876 5097 webserver.cc:460] Webserver started at http://127.0.98.1:40501/ using document root <none> and password file <none>
I20260619 09:01:22.757131 5097 fs_manager.cc:359] Metadata directory not provided
I20260619 09:01:22.757185 5097 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 09:01:22.757299 5097 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 09:01:22.758538 5097 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781859658134510-392-0/minicluster-data/ts-0/data/instance:
uuid: "8f3b3dfb3b1740a98126ff1656b767da"
format_stamp: "Formatted at 2026-06-19 09:01:22 on dist-test-slave-mtdr"
server_key: "ed065e17898fb665db950657a2ddf92b"
server_key_iv: "57c609ca97cf823e2045a959dafb35e2"
server_key_version: "encryptionkey@0"
I20260619 09:01:22.758924 5097 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781859658134510-392-0/minicluster-data/ts-0/wal/instance:
uuid: "8f3b3dfb3b1740a98126ff1656b767da"
format_stamp: "Formatted at 2026-06-19 09:01:22 on dist-test-slave-mtdr"
server_key: "ed065e17898fb665db950657a2ddf92b"
server_key_iv: "57c609ca97cf823e2045a959dafb35e2"
server_key_version: "encryptionkey@0"
I20260619 09:01:22.760319 5097 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 09:01:22.761055 5111 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 09:01:22.761190 5097 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 09:01:22.761262 5097 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781859658134510-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781859658134510-392-0/minicluster-data/ts-0/wal
uuid: "8f3b3dfb3b1740a98126ff1656b767da"
format_stamp: "Formatted at 2026-06-19 09:01:22 on dist-test-slave-mtdr"
server_key: "ed065e17898fb665db950657a2ddf92b"
server_key_iv: "57c609ca97cf823e2045a959dafb35e2"
server_key_version: "encryptionkey@0"
I20260619 09:01:22.761320 5097 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781859658134510-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781859658134510-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781859658134510-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 09:01:22.765808 5097 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 09:01:22.766003 5097 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 09:01:22.766093 5097 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 09:01:22.766244 5097 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 09:01:22.766469 5097 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 09:01:22.766495 5097 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 09:01:22.766515 5097 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 09:01:22.766530 5097 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 09:01:22.771634 5097 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40145
I20260619 09:01:22.771687 5224 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40145 every 8 connection(s)
I20260619 09:01:22.771943 5097 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781859658134510-392-0/minicluster-data/ts-0/data/info.pb
I20260619 09:01:22.773583 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ez5NB/build/release/bin/kudu as pid 5097
I20260619 09:01:22.773694 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781859658134510-392-0/minicluster-data/ts-0/wal/instance
I20260619 09:01:22.773846 392 external_mini_cluster.cc:1428] Setting key c72c743da3a59c4ff1bf2c7d88f7d301
I20260619 09:01:22.774734 5077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:22.772540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37819 (local address 127.0.98.62:37253)
0619 09:01:22.772592 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:22.772593 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:22.772821 (+ 228us) server_negotiation.cc:407] Connection header received
0619 09:01:22.773242 (+ 421us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:22.773244 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:22.773263 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:22.773312 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:01:22.774506 (+ 1194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:01:22.774507 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:01:22.774507 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 09:01:22.774512 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:01:22.774529 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 09:01:22.774534 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:01:22.774595 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0619 09:01:22.774652 (+ 57us) server_negotiation.cc:299] Negotiation successful
0619 09:01:22.774667 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260619 09:01:22.774777 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ez5NB/build/release/bin/kudu
/tmp/dist-test-task4ez5NB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781859658134510-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781859658134510-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781859658134510-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781859658134510-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37253
--builtin_ntp_servers=127.0.98.20:37451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260619 09:01:22.774889 5227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:22.772538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37253 (local address 127.0.98.1:37819)
0619 09:01:22.772702 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0619 09:01:22.772731 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0619 09:01:22.773158 (+ 427us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:01:22.773375 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:01:22.773378 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:01:22.773386 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0619 09:01:22.774347 (+ 961us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 09:01:22.774348 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 09:01:22.774383 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 09:01:22.774415 (+ 32us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 09:01:22.774593 (+ 178us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 09:01:22.774596 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 09:01:22.774598 (+ 2us) client_negotiation.cc:769] Sending connection context
0619 09:01:22.774645 (+ 47us) client_negotiation.cc:240] Negotiation successful
0619 09:01:22.774689 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":80,"threads_started":1}
I20260619 09:01:22.776081 5225 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37253
I20260619 09:01:22.776167 5225 heartbeater.cc:464] Registering TS with master...
I20260619 09:01:22.776297 5225 heartbeater.cc:511] Master 127.0.98.62:37253 requested a full tablet report, sending...
I20260619 09:01:22.776723 5037 ts_manager.cc:194] Registered new tserver with Master: 8f3b3dfb3b1740a98126ff1656b767da (127.0.98.1:40145)
W20260619 09:01:22.851920 5228 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 09:01:22.852111 5228 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 09:01:22.852131 5228 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 09:01:22.853260 5228 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 09:01:22.853302 5228 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 09:01:22.853325 5228 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 09:01:22.854655 5228 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781859658134510-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781859658134510-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781859658134510-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37253
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5228
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781859658134510-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 09:01:22.854895 5228 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 09:01:22.855161 5228 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 09:01:22.859660 5233 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:01:22.859683 5236 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:01:22.859666 5234 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:01:22.859958 5228 server_base.cc:1034] running on GCE node
I20260619 09:01:22.860132 5228 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 09:01:22.860351 5228 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 09:01:22.861500 5228 hybrid_clock.cc:648] HybridClock initialized: now 1781859682861482 us; error 27 us; skew 500 ppm
I20260619 09:01:22.862726 5228 webserver.cc:460] Webserver started at http://127.0.98.2:35851/ using document root <none> and password file <none>
I20260619 09:01:22.862955 5228 fs_manager.cc:359] Metadata directory not provided
I20260619 09:01:22.863004 5228 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 09:01:22.863125 5228 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 09:01:22.864096 5228 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781859658134510-392-0/minicluster-data/ts-1/data/instance:
uuid: "58447d2f949d44ff9293003bc624af4a"
format_stamp: "Formatted at 2026-06-19 09:01:22 on dist-test-slave-mtdr"
server_key: "a1aca629aaafd95c3472f4e194e758b0"
server_key_iv: "b0265ec05ac14de058feb997e20ba23c"
server_key_version: "encryptionkey@0"
I20260619 09:01:22.864414 5228 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781859658134510-392-0/minicluster-data/ts-1/wal/instance:
uuid: "58447d2f949d44ff9293003bc624af4a"
format_stamp: "Formatted at 2026-06-19 09:01:22 on dist-test-slave-mtdr"
server_key: "a1aca629aaafd95c3472f4e194e758b0"
server_key_iv: "b0265ec05ac14de058feb997e20ba23c"
server_key_version: "encryptionkey@0"
I20260619 09:01:22.865607 5228 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 09:01:22.866349 5242 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 09:01:22.866501 5228 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 09:01:22.866580 5228 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781859658134510-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781859658134510-392-0/minicluster-data/ts-1/wal
uuid: "58447d2f949d44ff9293003bc624af4a"
format_stamp: "Formatted at 2026-06-19 09:01:22 on dist-test-slave-mtdr"
server_key: "a1aca629aaafd95c3472f4e194e758b0"
server_key_iv: "b0265ec05ac14de058feb997e20ba23c"
server_key_version: "encryptionkey@0"
I20260619 09:01:22.866647 5228 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781859658134510-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781859658134510-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781859658134510-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 09:01:22.869721 5228 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 09:01:22.869915 5228 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 09:01:22.869999 5228 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 09:01:22.870136 5228 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 09:01:22.870357 5228 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 09:01:22.870383 5228 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 09:01:22.870400 5228 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 09:01:22.870419 5228 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 09:01:22.875574 5228 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39999
I20260619 09:01:22.875643 5355 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39999 every 8 connection(s)
I20260619 09:01:22.875897 5228 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781859658134510-392-0/minicluster-data/ts-1/data/info.pb
I20260619 09:01:22.878757 5077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:22.876568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47159 (local address 127.0.98.62:37253)
0619 09:01:22.876636 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:22.876639 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:22.876835 (+ 196us) server_negotiation.cc:407] Connection header received
0619 09:01:22.877311 (+ 476us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:22.877312 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:22.877319 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:22.877384 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:01:22.878485 (+ 1101us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:01:22.878486 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:01:22.878487 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 09:01:22.878491 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:01:22.878507 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 09:01:22.878511 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:01:22.878569 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0619 09:01:22.878678 (+ 109us) server_negotiation.cc:299] Negotiation successful
0619 09:01:22.878691 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260619 09:01:22.878943 5358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:22.876540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37253 (local address 127.0.98.2:47159)
0619 09:01:22.876722 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0619 09:01:22.876753 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0619 09:01:22.877212 (+ 459us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:01:22.877457 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:01:22.877460 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:01:22.877468 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0619 09:01:22.878333 (+ 865us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 09:01:22.878334 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 09:01:22.878361 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 09:01:22.878387 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 09:01:22.878580 (+ 193us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 09:01:22.878585 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 09:01:22.878588 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 09:01:22.878680 (+ 92us) client_negotiation.cc:240] Negotiation successful
0619 09:01:22.878739 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":82,"threads_started":1}
I20260619 09:01:22.879192 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ez5NB/build/release/bin/kudu as pid 5228
I20260619 09:01:22.879271 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781859658134510-392-0/minicluster-data/ts-1/wal/instance
I20260619 09:01:22.879360 5356 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37253
I20260619 09:01:22.879441 5356 heartbeater.cc:464] Registering TS with master...
I20260619 09:01:22.879448 392 external_mini_cluster.cc:1428] Setting key 8b868c038085f3761e58decbbecd729a
I20260619 09:01:22.879575 5356 heartbeater.cc:511] Master 127.0.98.62:37253 requested a full tablet report, sending...
I20260619 09:01:22.879899 5036 ts_manager.cc:194] Registered new tserver with Master: 58447d2f949d44ff9293003bc624af4a (127.0.98.2:39999)
I20260619 09:01:22.880235 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ez5NB/build/release/bin/kudu
/tmp/dist-test-task4ez5NB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781859658134510-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781859658134510-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781859658134510-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781859658134510-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37253
--builtin_ntp_servers=127.0.98.20:37451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260619 09:01:22.957239 5359 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 09:01:22.957437 5359 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 09:01:22.957455 5359 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 09:01:22.958555 5359 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 09:01:22.958592 5359 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 09:01:22.958616 5359 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 09:01:22.959930 5359 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781859658134510-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781859658134510-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781859658134510-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37253
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5359
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781859658134510-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 09:01:22.960145 5359 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 09:01:22.960414 5359 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 09:01:22.965123 5365 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:01:22.965143 5364 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:01:22.965127 5367 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:01:22.965260 5359 server_base.cc:1034] running on GCE node
I20260619 09:01:22.965658 5359 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 09:01:22.965898 5359 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 09:01:22.967034 5359 hybrid_clock.cc:648] HybridClock initialized: now 1781859682967019 us; error 26 us; skew 500 ppm
I20260619 09:01:22.968124 5359 webserver.cc:460] Webserver started at http://127.0.98.3:39815/ using document root <none> and password file <none>
I20260619 09:01:22.968335 5359 fs_manager.cc:359] Metadata directory not provided
I20260619 09:01:22.968382 5359 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 09:01:22.968503 5359 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 09:01:22.969578 5359 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781859658134510-392-0/minicluster-data/ts-2/data/instance:
uuid: "97fd1f60950a4384abccd959deed07de"
format_stamp: "Formatted at 2026-06-19 09:01:22 on dist-test-slave-mtdr"
server_key: "98bfc76690e08facfc37b65ac33619a4"
server_key_iv: "7310e203687dee70f8d425fccc063016"
server_key_version: "encryptionkey@0"
I20260619 09:01:22.969902 5359 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781859658134510-392-0/minicluster-data/ts-2/wal/instance:
uuid: "97fd1f60950a4384abccd959deed07de"
format_stamp: "Formatted at 2026-06-19 09:01:22 on dist-test-slave-mtdr"
server_key: "98bfc76690e08facfc37b65ac33619a4"
server_key_iv: "7310e203687dee70f8d425fccc063016"
server_key_version: "encryptionkey@0"
I20260619 09:01:22.971095 5359 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260619 09:01:22.971748 5373 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 09:01:22.971894 5359 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 09:01:22.971968 5359 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781859658134510-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781859658134510-392-0/minicluster-data/ts-2/wal
uuid: "97fd1f60950a4384abccd959deed07de"
format_stamp: "Formatted at 2026-06-19 09:01:22 on dist-test-slave-mtdr"
server_key: "98bfc76690e08facfc37b65ac33619a4"
server_key_iv: "7310e203687dee70f8d425fccc063016"
server_key_version: "encryptionkey@0"
I20260619 09:01:22.972026 5359 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781859658134510-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781859658134510-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781859658134510-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 09:01:22.975239 5359 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 09:01:22.975461 5359 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 09:01:22.975570 5359 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 09:01:22.975740 5359 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 09:01:22.976004 5359 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 09:01:22.976034 5359 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 09:01:22.976059 5359 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 09:01:22.976083 5359 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 09:01:22.981132 5359 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40775
I20260619 09:01:22.981187 5486 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40775 every 8 connection(s)
I20260619 09:01:22.981438 5359 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781859658134510-392-0/minicluster-data/ts-2/data/info.pb
I20260619 09:01:22.984050 5077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:22.982076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52159 (local address 127.0.98.62:37253)
0619 09:01:22.982130 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:22.982132 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:22.982320 (+ 188us) server_negotiation.cc:407] Connection header received
0619 09:01:22.982730 (+ 410us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:22.982731 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:22.982735 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:22.982771 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:01:22.983844 (+ 1073us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:01:22.983845 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:01:22.983845 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 09:01:22.983850 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:01:22.983864 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 09:01:22.983868 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:01:22.983924 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0619 09:01:22.983979 (+ 55us) server_negotiation.cc:299] Negotiation successful
0619 09:01:22.983992 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260619 09:01:22.984202 5489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:22.982047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37253 (local address 127.0.98.3:52159)
0619 09:01:22.982205 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0619 09:01:22.982241 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0619 09:01:22.982645 (+ 404us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:01:22.982856 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:01:22.982860 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:01:22.982869 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0619 09:01:22.983660 (+ 791us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 09:01:22.983660 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0619 09:01:22.983712 (+ 52us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 09:01:22.983744 (+ 32us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 09:01:22.983922 (+ 178us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 09:01:22.983925 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 09:01:22.983927 (+ 2us) client_negotiation.cc:769] Sending connection context
0619 09:01:22.983979 (+ 52us) client_negotiation.cc:240] Negotiation successful
0619 09:01:22.984023 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":72,"threads_started":1}
I20260619 09:01:22.984321 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ez5NB/build/release/bin/kudu as pid 5359
I20260619 09:01:22.984414 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781859658134510-392-0/minicluster-data/ts-2/wal/instance
I20260619 09:01:22.984584 392 external_mini_cluster.cc:1428] Setting key b295ed4cbacaa586d61d9c70e91c338e
I20260619 09:01:22.984599 5487 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37253
I20260619 09:01:22.984673 5487 heartbeater.cc:464] Registering TS with master...
I20260619 09:01:22.984820 5487 heartbeater.cc:511] Master 127.0.98.62:37253 requested a full tablet report, sending...
I20260619 09:01:22.985205 5037 ts_manager.cc:194] Registered new tserver with Master: 97fd1f60950a4384abccd959deed07de (127.0.98.3:40775)
I20260619 09:01:22.996173 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 09:01:22.998329 5077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:22.997414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51764 (local address 127.0.98.62:37253)
0619 09:01:22.997484 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:22.997486 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:22.997542 (+ 56us) server_negotiation.cc:407] Connection header received
0619 09:01:22.997602 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:22.997603 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:22.997608 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:22.997654 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:01:22.998126 (+ 472us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:01:22.998127 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:01:22.998127 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 09:01:22.998131 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:01:22.998146 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 09:01:22.998150 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:01:22.998196 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0619 09:01:22.998248 (+ 52us) server_negotiation.cc:299] Negotiation successful
0619 09:01:22.998261 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 09:01:22.999858 5037 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51764:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 09:01:23.001299 5037 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 09:01:23.004982 5495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:23.003515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40775 (local address 127.0.0.1:35242)
0619 09:01:23.003783 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0619 09:01:23.003793 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 09:01:23.003852 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:01:23.004111 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:01:23.004116 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:01:23.004123 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0619 09:01:23.004586 (+ 463us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 09:01:23.004586 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0619 09:01:23.004610 (+ 24us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 09:01:23.004635 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 09:01:23.004843 (+ 208us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 09:01:23.004849 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 09:01:23.004851 (+ 2us) client_negotiation.cc:769] Sending connection context
0619 09:01:23.004907 (+ 56us) client_negotiation.cc:240] Negotiation successful
0619 09:01:23.004919 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":56,"threads_started":1}
I20260619 09:01:23.005165 5500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:23.003629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35242 (local address 127.0.98.3:40775)
0619 09:01:23.003943 (+ 314us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:23.003947 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:23.003956 (+ 9us) server_negotiation.cc:407] Connection header received
0619 09:01:23.003978 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:23.003980 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:23.003989 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:23.004068 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:01:23.004696 (+ 628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:01:23.004703 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:01:23.004703 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 09:01:23.004707 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:01:23.004738 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 09:01:23.004769 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:01:23.004819 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0619 09:01:23.005072 (+ 253us) server_negotiation.cc:299] Negotiation successful
0619 09:01:23.005088 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":277,"thread_start_us":79,"threads_started":1}
I20260619 09:01:23.005157 5496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:23.003303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39999 (local address 127.0.0.1:34940)
0619 09:01:23.003478 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0619 09:01:23.003513 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0619 09:01:23.003566 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:01:23.004415 (+ 849us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:01:23.004416 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:01:23.004419 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0619 09:01:23.004818 (+ 399us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 09:01:23.004818 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0619 09:01:23.004832 (+ 14us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 09:01:23.004840 (+ 8us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 09:01:23.005026 (+ 186us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 09:01:23.005027 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 09:01:23.005027 (+ 0us) client_negotiation.cc:769] Sending connection context
0619 09:01:23.005070 (+ 43us) client_negotiation.cc:240] Negotiation successful
0619 09:01:23.005081 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":52,"threads_started":1}
I20260619 09:01:23.005369 5499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:23.003633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34940 (local address 127.0.98.2:39999)
0619 09:01:23.003774 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:23.003777 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:23.003785 (+ 8us) server_negotiation.cc:407] Connection header received
0619 09:01:23.003805 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:23.003806 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:23.003814 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:23.004402 (+ 588us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:01:23.004913 (+ 511us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:01:23.004919 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:01:23.004919 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 09:01:23.004923 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:01:23.004950 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 09:01:23.004971 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:01:23.005014 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0619 09:01:23.005296 (+ 282us) server_negotiation.cc:299] Negotiation successful
0619 09:01:23.005311 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":74,"threads_started":1}
I20260619 09:01:23.005470 5498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:23.003317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40145 (local address 127.0.0.1:53984)
0619 09:01:23.003783 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0619 09:01:23.003793 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 09:01:23.003851 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:01:23.004359 (+ 508us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:01:23.004360 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:01:23.004363 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0619 09:01:23.004672 (+ 309us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 09:01:23.004673 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 09:01:23.004679 (+ 6us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 09:01:23.004691 (+ 12us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 09:01:23.005041 (+ 350us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 09:01:23.005042 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 09:01:23.005043 (+ 1us) client_negotiation.cc:769] Sending connection context
0619 09:01:23.005089 (+ 46us) client_negotiation.cc:240] Negotiation successful
0619 09:01:23.005102 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":101,"threads_started":1}
I20260619 09:01:23.005488 5497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:23.003420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53984 (local address 127.0.98.1:40145)
0619 09:01:23.003582 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:23.003588 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:23.003979 (+ 391us) server_negotiation.cc:407] Connection header received
0619 09:01:23.004006 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:23.004009 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:23.004018 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:23.004081 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:01:23.004767 (+ 686us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:01:23.004773 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:01:23.004773 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 09:01:23.004778 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:01:23.004812 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 09:01:23.004835 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:01:23.004903 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0619 09:01:23.005193 (+ 290us) server_negotiation.cc:299] Negotiation successful
0619 09:01:23.005404 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":72,"threads_started":1}
I20260619 09:01:23.005810 5421 tablet_service.cc:1467] Processing CreateTablet for tablet 6651201b8e5b4963842ce583664ea750 (DEFAULT_TABLE table=test-table [id=568471bb209640b7be9a0ce27b5c6ca8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 09:01:23.005996 5290 tablet_service.cc:1467] Processing CreateTablet for tablet 6651201b8e5b4963842ce583664ea750 (DEFAULT_TABLE table=test-table [id=568471bb209640b7be9a0ce27b5c6ca8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 09:01:23.006148 5421 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6651201b8e5b4963842ce583664ea750. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 09:01:23.006261 5290 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6651201b8e5b4963842ce583664ea750. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 09:01:23.007148 5159 tablet_service.cc:1467] Processing CreateTablet for tablet 6651201b8e5b4963842ce583664ea750 (DEFAULT_TABLE table=test-table [id=568471bb209640b7be9a0ce27b5c6ca8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 09:01:23.007426 5159 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6651201b8e5b4963842ce583664ea750. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 09:01:23.009452 5502 tablet_bootstrap.cc:492] T 6651201b8e5b4963842ce583664ea750 P 97fd1f60950a4384abccd959deed07de: Bootstrap starting.
I20260619 09:01:23.010001 5503 tablet_bootstrap.cc:492] T 6651201b8e5b4963842ce583664ea750 P 8f3b3dfb3b1740a98126ff1656b767da: Bootstrap starting.
I20260619 09:01:23.010116 5502 tablet_bootstrap.cc:654] T 6651201b8e5b4963842ce583664ea750 P 97fd1f60950a4384abccd959deed07de: Neither blocks nor log segments found. Creating new log.
I20260619 09:01:23.010149 5501 tablet_bootstrap.cc:492] T 6651201b8e5b4963842ce583664ea750 P 58447d2f949d44ff9293003bc624af4a: Bootstrap starting.
I20260619 09:01:23.010422 5502 log.cc:824] T 6651201b8e5b4963842ce583664ea750 P 97fd1f60950a4384abccd959deed07de: Log is configured to *not* fsync() on all Append() calls
I20260619 09:01:23.010598 5503 tablet_bootstrap.cc:654] T 6651201b8e5b4963842ce583664ea750 P 8f3b3dfb3b1740a98126ff1656b767da: Neither blocks nor log segments found. Creating new log.
I20260619 09:01:23.010740 5501 tablet_bootstrap.cc:654] T 6651201b8e5b4963842ce583664ea750 P 58447d2f949d44ff9293003bc624af4a: Neither blocks nor log segments found. Creating new log.
I20260619 09:01:23.010910 5503 log.cc:824] T 6651201b8e5b4963842ce583664ea750 P 8f3b3dfb3b1740a98126ff1656b767da: Log is configured to *not* fsync() on all Append() calls
I20260619 09:01:23.011057 5501 log.cc:824] T 6651201b8e5b4963842ce583664ea750 P 58447d2f949d44ff9293003bc624af4a: Log is configured to *not* fsync() on all Append() calls
I20260619 09:01:23.011212 5502 tablet_bootstrap.cc:492] T 6651201b8e5b4963842ce583664ea750 P 97fd1f60950a4384abccd959deed07de: No bootstrap required, opened a new log
I20260619 09:01:23.011283 5502 ts_tablet_manager.cc:1397] T 6651201b8e5b4963842ce583664ea750 P 97fd1f60950a4384abccd959deed07de: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 09:01:23.011583 5503 tablet_bootstrap.cc:492] T 6651201b8e5b4963842ce583664ea750 P 8f3b3dfb3b1740a98126ff1656b767da: No bootstrap required, opened a new log
I20260619 09:01:23.011637 5503 ts_tablet_manager.cc:1397] T 6651201b8e5b4963842ce583664ea750 P 8f3b3dfb3b1740a98126ff1656b767da: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 09:01:23.011788 5501 tablet_bootstrap.cc:492] T 6651201b8e5b4963842ce583664ea750 P 58447d2f949d44ff9293003bc624af4a: No bootstrap required, opened a new log
I20260619 09:01:23.011857 5501 ts_tablet_manager.cc:1397] T 6651201b8e5b4963842ce583664ea750 P 58447d2f949d44ff9293003bc624af4a: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260619 09:01:23.013386 5503 raft_consensus.cc:348] T 6651201b8e5b4963842ce583664ea750 P 8f3b3dfb3b1740a98126ff1656b767da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f3b3dfb3b1740a98126ff1656b767da" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40145 } } peers { permanent_uuid: "58447d2f949d44ff9293003bc624af4a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39999 } } peers { permanent_uuid: "97fd1f60950a4384abccd959deed07de" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40775 } }
I20260619 09:01:23.013513 5503 raft_consensus.cc:374] T 6651201b8e5b4963842ce583664ea750 P 8f3b3dfb3b1740a98126ff1656b767da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 09:01:23.013543 5503 raft_consensus.cc:729] T 6651201b8e5b4963842ce583664ea750 P 8f3b3dfb3b1740a98126ff1656b767da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f3b3dfb3b1740a98126ff1656b767da, State: Initialized, Role: FOLLOWER
I20260619 09:01:23.013651 5503 consensus_queue.cc:260] T 6651201b8e5b4963842ce583664ea750 P 8f3b3dfb3b1740a98126ff1656b767da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f3b3dfb3b1740a98126ff1656b767da" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40145 } } peers { permanent_uuid: "58447d2f949d44ff9293003bc624af4a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39999 } } peers { permanent_uuid: "97fd1f60950a4384abccd959deed07de" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40775 } }
I20260619 09:01:23.013684 5502 raft_consensus.cc:348] T 6651201b8e5b4963842ce583664ea750 P 97fd1f60950a4384abccd959deed07de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f3b3dfb3b1740a98126ff1656b767da" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40145 } } peers { permanent_uuid: "58447d2f949d44ff9293003bc624af4a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39999 } } peers { permanent_uuid: "97fd1f60950a4384abccd959deed07de" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40775 } }
I20260619 09:01:23.013864 5502 raft_consensus.cc:374] T 6651201b8e5b4963842ce583664ea750 P 97fd1f60950a4384abccd959deed07de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 09:01:23.013888 5503 ts_tablet_manager.cc:1428] T 6651201b8e5b4963842ce583664ea750 P 8f3b3dfb3b1740a98126ff1656b767da: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260619 09:01:23.013904 5502 raft_consensus.cc:729] T 6651201b8e5b4963842ce583664ea750 P 97fd1f60950a4384abccd959deed07de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97fd1f60950a4384abccd959deed07de, State: Initialized, Role: FOLLOWER
I20260619 09:01:23.014015 5225 heartbeater.cc:503] Master 127.0.98.62:37253 was elected leader, sending a full tablet report...
I20260619 09:01:23.014034 5502 consensus_queue.cc:260] T 6651201b8e5b4963842ce583664ea750 P 97fd1f60950a4384abccd959deed07de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f3b3dfb3b1740a98126ff1656b767da" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40145 } } peers { permanent_uuid: "58447d2f949d44ff9293003bc624af4a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39999 } } peers { permanent_uuid: "97fd1f60950a4384abccd959deed07de" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40775 } }
I20260619 09:01:23.014267 5502 ts_tablet_manager.cc:1428] T 6651201b8e5b4963842ce583664ea750 P 97fd1f60950a4384abccd959deed07de: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260619 09:01:23.014256 5501 raft_consensus.cc:348] T 6651201b8e5b4963842ce583664ea750 P 58447d2f949d44ff9293003bc624af4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f3b3dfb3b1740a98126ff1656b767da" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40145 } } peers { permanent_uuid: "58447d2f949d44ff9293003bc624af4a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39999 } } peers { permanent_uuid: "97fd1f60950a4384abccd959deed07de" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40775 } }
I20260619 09:01:23.014406 5501 raft_consensus.cc:374] T 6651201b8e5b4963842ce583664ea750 P 58447d2f949d44ff9293003bc624af4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 09:01:23.014456 5501 raft_consensus.cc:729] T 6651201b8e5b4963842ce583664ea750 P 58447d2f949d44ff9293003bc624af4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58447d2f949d44ff9293003bc624af4a, State: Initialized, Role: FOLLOWER
I20260619 09:01:23.014454 5487 heartbeater.cc:503] Master 127.0.98.62:37253 was elected leader, sending a full tablet report...
I20260619 09:01:23.014551 5501 consensus_queue.cc:260] T 6651201b8e5b4963842ce583664ea750 P 58447d2f949d44ff9293003bc624af4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f3b3dfb3b1740a98126ff1656b767da" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40145 } } peers { permanent_uuid: "58447d2f949d44ff9293003bc624af4a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39999 } } peers { permanent_uuid: "97fd1f60950a4384abccd959deed07de" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40775 } }
I20260619 09:01:23.014783 5501 ts_tablet_manager.cc:1428] T 6651201b8e5b4963842ce583664ea750 P 58447d2f949d44ff9293003bc624af4a: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260619 09:01:23.014904 5356 heartbeater.cc:503] Master 127.0.98.62:37253 was elected leader, sending a full tablet report...
W20260619 09:01:23.022395 5226 tablet.cc:2369] T 6651201b8e5b4963842ce583664ea750 P 8f3b3dfb3b1740a98126ff1656b767da: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 09:01:23.116941 5508 raft_consensus.cc:482] T 6651201b8e5b4963842ce583664ea750 P 97fd1f60950a4384abccd959deed07de [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 09:01:23.117061 5508 raft_consensus.cc:504] T 6651201b8e5b4963842ce583664ea750 P 97fd1f60950a4384abccd959deed07de [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f3b3dfb3b1740a98126ff1656b767da" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40145 } } peers { permanent_uuid: "58447d2f949d44ff9293003bc624af4a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39999 } } peers { permanent_uuid: "97fd1f60950a4384abccd959deed07de" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40775 } }
I20260619 09:01:23.117455 5508 leader_election.cc:288] T 6651201b8e5b4963842ce583664ea750 P 97fd1f60950a4384abccd959deed07de [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f3b3dfb3b1740a98126ff1656b767da (127.0.98.1:40145), 58447d2f949d44ff9293003bc624af4a (127.0.98.2:39999)
I20260619 09:01:23.118569 5497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:23.117657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45629 (local address 127.0.98.1:40145)
0619 09:01:23.117703 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:23.117705 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:23.117737 (+ 32us) server_negotiation.cc:407] Connection header received
0619 09:01:23.117761 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:23.117762 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:23.117768 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:23.117814 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:01:23.118374 (+ 560us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:01:23.118376 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:01:23.118376 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 09:01:23.118380 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:01:23.118395 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 09:01:23.118399 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:01:23.118448 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0619 09:01:23.118497 (+ 49us) server_negotiation.cc:299] Negotiation successful
0619 09:01:23.118509 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260619 09:01:23.118563 5489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:23.117599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40145 (local address 127.0.98.3:45629)
0619 09:01:23.117655 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0619 09:01:23.117665 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 09:01:23.117701 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:01:23.117868 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:01:23.117870 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:01:23.117872 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0619 09:01:23.118289 (+ 417us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 09:01:23.118289 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0619 09:01:23.118295 (+ 6us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 09:01:23.118305 (+ 10us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 09:01:23.118448 (+ 143us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 09:01:23.118450 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 09:01:23.118450 (+ 0us) client_negotiation.cc:769] Sending connection context
0619 09:01:23.118487 (+ 37us) client_negotiation.cc:240] Negotiation successful
0619 09:01:23.118499 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260619 09:01:23.118732 5510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:23.117616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39999 (local address 127.0.98.3:58371)
0619 09:01:23.117816 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0619 09:01:23.117826 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 09:01:23.117885 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:01:23.117997 (+ 112us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:01:23.117998 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:01:23.118000 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0619 09:01:23.118407 (+ 407us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 09:01:23.118408 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 09:01:23.118415 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 09:01:23.118427 (+ 12us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 09:01:23.118581 (+ 154us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 09:01:23.118582 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 09:01:23.118582 (+ 0us) client_negotiation.cc:769] Sending connection context
0619 09:01:23.118628 (+ 46us) client_negotiation.cc:240] Negotiation successful
0619 09:01:23.118640 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":114,"threads_started":1}
I20260619 09:01:23.118845 5499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:23.117675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58371 (local address 127.0.98.2:39999)
0619 09:01:23.117713 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:23.117715 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:23.117886 (+ 171us) server_negotiation.cc:407] Connection header received
0619 09:01:23.117937 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:23.117938 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:23.117943 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:23.117993 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:01:23.118502 (+ 509us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:01:23.118504 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:01:23.118504 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 09:01:23.118509 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:01:23.118525 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 09:01:23.118530 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:01:23.118579 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0619 09:01:23.118732 (+ 153us) server_negotiation.cc:299] Negotiation successful
0619 09:01:23.118745 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260619 09:01:23.118921 5179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6651201b8e5b4963842ce583664ea750" candidate_uuid: "97fd1f60950a4384abccd959deed07de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f3b3dfb3b1740a98126ff1656b767da" is_pre_election: true
I20260619 09:01:23.119072 5179 raft_consensus.cc:2454] T 6651201b8e5b4963842ce583664ea750 P 8f3b3dfb3b1740a98126ff1656b767da [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97fd1f60950a4384abccd959deed07de in term 0.
I20260619 09:01:23.119289 5375 leader_election.cc:302] T 6651201b8e5b4963842ce583664ea750 P 97fd1f60950a4384abccd959deed07de [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8f3b3dfb3b1740a98126ff1656b767da, 97fd1f60950a4384abccd959deed07de; no voters:
I20260619 09:01:23.119313 5310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6651201b8e5b4963842ce583664ea750" candidate_uuid: "97fd1f60950a4384abccd959deed07de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58447d2f949d44ff9293003bc624af4a" is_pre_election: true
I20260619 09:01:23.119426 5310 raft_consensus.cc:2454] T 6651201b8e5b4963842ce583664ea750 P 58447d2f949d44ff9293003bc624af4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97fd1f60950a4384abccd959deed07de in term 0.
I20260619 09:01:23.119485 5508 raft_consensus.cc:2781] T 6651201b8e5b4963842ce583664ea750 P 97fd1f60950a4384abccd959deed07de [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 09:01:23.119549 5508 raft_consensus.cc:482] T 6651201b8e5b4963842ce583664ea750 P 97fd1f60950a4384abccd959deed07de [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 09:01:23.119570 5508 raft_consensus.cc:3037] T 6651201b8e5b4963842ce583664ea750 P 97fd1f60950a4384abccd959deed07de [term 0 FOLLOWER]: Advancing to term 1
I20260619 09:01:23.120219 5508 raft_consensus.cc:504] T 6651201b8e5b4963842ce583664ea750 P 97fd1f60950a4384abccd959deed07de [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f3b3dfb3b1740a98126ff1656b767da" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40145 } } peers { permanent_uuid: "58447d2f949d44ff9293003bc624af4a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39999 } } peers { permanent_uuid: "97fd1f60950a4384abccd959deed07de" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40775 } }
I20260619 09:01:23.120357 5508 leader_election.cc:288] T 6651201b8e5b4963842ce583664ea750 P 97fd1f60950a4384abccd959deed07de [CANDIDATE]: Term 1 election: Requested vote from peers 8f3b3dfb3b1740a98126ff1656b767da (127.0.98.1:40145), 58447d2f949d44ff9293003bc624af4a (127.0.98.2:39999)
I20260619 09:01:23.120493 5179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6651201b8e5b4963842ce583664ea750" candidate_uuid: "97fd1f60950a4384abccd959deed07de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f3b3dfb3b1740a98126ff1656b767da"
I20260619 09:01:23.120513 5310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6651201b8e5b4963842ce583664ea750" candidate_uuid: "97fd1f60950a4384abccd959deed07de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58447d2f949d44ff9293003bc624af4a"
I20260619 09:01:23.120568 5179 raft_consensus.cc:3037] T 6651201b8e5b4963842ce583664ea750 P 8f3b3dfb3b1740a98126ff1656b767da [term 0 FOLLOWER]: Advancing to term 1
I20260619 09:01:23.120592 5310 raft_consensus.cc:3037] T 6651201b8e5b4963842ce583664ea750 P 58447d2f949d44ff9293003bc624af4a [term 0 FOLLOWER]: Advancing to term 1
I20260619 09:01:23.121416 5179 raft_consensus.cc:2454] T 6651201b8e5b4963842ce583664ea750 P 8f3b3dfb3b1740a98126ff1656b767da [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97fd1f60950a4384abccd959deed07de in term 1.
I20260619 09:01:23.121483 5310 raft_consensus.cc:2454] T 6651201b8e5b4963842ce583664ea750 P 58447d2f949d44ff9293003bc624af4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97fd1f60950a4384abccd959deed07de in term 1.
I20260619 09:01:23.121590 5375 leader_election.cc:302] T 6651201b8e5b4963842ce583664ea750 P 97fd1f60950a4384abccd959deed07de [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8f3b3dfb3b1740a98126ff1656b767da, 97fd1f60950a4384abccd959deed07de; no voters:
I20260619 09:01:23.121666 5508 raft_consensus.cc:2781] T 6651201b8e5b4963842ce583664ea750 P 97fd1f60950a4384abccd959deed07de [term 1 FOLLOWER]: Leader election won for term 1
I20260619 09:01:23.121809 5508 raft_consensus.cc:686] T 6651201b8e5b4963842ce583664ea750 P 97fd1f60950a4384abccd959deed07de [term 1 LEADER]: Becoming Leader. State: Replica: 97fd1f60950a4384abccd959deed07de, State: Running, Role: LEADER
I20260619 09:01:23.121884 5508 consensus_queue.cc:237] T 6651201b8e5b4963842ce583664ea750 P 97fd1f60950a4384abccd959deed07de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f3b3dfb3b1740a98126ff1656b767da" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40145 } } peers { permanent_uuid: "58447d2f949d44ff9293003bc624af4a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39999 } } peers { permanent_uuid: "97fd1f60950a4384abccd959deed07de" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40775 } }
I20260619 09:01:23.122596 5036 catalog_manager.cc:5495] T 6651201b8e5b4963842ce583664ea750 P 97fd1f60950a4384abccd959deed07de reported cstate change: term changed from 0 to 1, leader changed from <none> to 97fd1f60950a4384abccd959deed07de (127.0.98.3). New cstate: current_term: 1 leader_uuid: "97fd1f60950a4384abccd959deed07de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f3b3dfb3b1740a98126ff1656b767da" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40145 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "58447d2f949d44ff9293003bc624af4a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39999 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "97fd1f60950a4384abccd959deed07de" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40775 } health_report { overall_health: HEALTHY } } }
W20260619 09:01:23.126369 5357 tablet.cc:2369] T 6651201b8e5b4963842ce583664ea750 P 58447d2f949d44ff9293003bc624af4a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 09:01:23.149734 5500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:23.148796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35254 (local address 127.0.98.3:40775)
0619 09:01:23.148845 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:23.148846 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:23.148859 (+ 13us) server_negotiation.cc:407] Connection header received
0619 09:01:23.148889 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:23.148890 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:23.148901 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:23.148958 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:01:23.149542 (+ 584us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:01:23.149544 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:01:23.149544 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 09:01:23.149548 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:01:23.149565 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 09:01:23.149569 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:01:23.149623 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0619 09:01:23.149669 (+ 46us) server_negotiation.cc:299] Negotiation successful
0619 09:01:23.149681 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260619 09:01:23.151413 5179 raft_consensus.cc:1261] T 6651201b8e5b4963842ce583664ea750 P 8f3b3dfb3b1740a98126ff1656b767da [term 1 FOLLOWER]: Refusing update from remote peer 97fd1f60950a4384abccd959deed07de: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 09:01:23.151427 5310 raft_consensus.cc:1261] T 6651201b8e5b4963842ce583664ea750 P 58447d2f949d44ff9293003bc624af4a [term 1 FOLLOWER]: Refusing update from remote peer 97fd1f60950a4384abccd959deed07de: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 09:01:23.151748 5508 consensus_queue.cc:1035] T 6651201b8e5b4963842ce583664ea750 P 97fd1f60950a4384abccd959deed07de [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f3b3dfb3b1740a98126ff1656b767da" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 09:01:23.151942 5511 consensus_queue.cc:1035] T 6651201b8e5b4963842ce583664ea750 P 97fd1f60950a4384abccd959deed07de [LEADER]: Connected to new peer: Peer: permanent_uuid: "58447d2f949d44ff9293003bc624af4a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 09:01:23.154474 5515 mvcc.cc:205] Tried to move back new op lower bound from 7298497262185304064 to 7298497262067429376. Current Snapshot: MvccSnapshot[applied={T|T < 7298497262185304064}]
I20260619 09:01:23.155010 5516 mvcc.cc:205] Tried to move back new op lower bound from 7298497262185304064 to 7298497262067429376. Current Snapshot: MvccSnapshot[applied={T|T < 7298497262185304064}]
I20260619 09:01:23.156309 5037 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51764:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 09:01:23.156402 5037 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51764:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 09:01:23.157172 5037 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bc3a10589f9c476485a0394e81062306: Sending DeleteTablet for 3 replicas of tablet 6651201b8e5b4963842ce583664ea750
I20260619 09:01:23.157536 5290 tablet_service.cc:1514] Processing DeleteTablet for tablet 6651201b8e5b4963842ce583664ea750 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 09:01:23 UTC) from {username='slave'} at 127.0.0.1:34940
I20260619 09:01:23.157567 5159 tablet_service.cc:1514] Processing DeleteTablet for tablet 6651201b8e5b4963842ce583664ea750 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 09:01:23 UTC) from {username='slave'} at 127.0.0.1:53984
I20260619 09:01:23.157567 5421 tablet_service.cc:1514] Processing DeleteTablet for tablet 6651201b8e5b4963842ce583664ea750 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 09:01:23 UTC) from {username='slave'} at 127.0.0.1:35242
I20260619 09:01:23.157775 5526 tablet_replica.cc:330] T 6651201b8e5b4963842ce583664ea750 P 8f3b3dfb3b1740a98126ff1656b767da: stopping tablet replica
I20260619 09:01:23.157811 5524 tablet_replica.cc:330] T 6651201b8e5b4963842ce583664ea750 P 58447d2f949d44ff9293003bc624af4a: stopping tablet replica
I20260619 09:01:23.157871 5526 raft_consensus.cc:2229] T 6651201b8e5b4963842ce583664ea750 P 8f3b3dfb3b1740a98126ff1656b767da [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 09:01:23.157907 5524 raft_consensus.cc:2229] T 6651201b8e5b4963842ce583664ea750 P 58447d2f949d44ff9293003bc624af4a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 09:01:23.157963 5526 raft_consensus.cc:2258] T 6651201b8e5b4963842ce583664ea750 P 8f3b3dfb3b1740a98126ff1656b767da [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 09:01:23.158015 5524 raft_consensus.cc:2258] T 6651201b8e5b4963842ce583664ea750 P 58447d2f949d44ff9293003bc624af4a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 09:01:23.158067 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ez5NB/build/release/bin/kudu with pid 5097
I20260619 09:01:23.158564 5525 tablet_replica.cc:330] T 6651201b8e5b4963842ce583664ea750 P 97fd1f60950a4384abccd959deed07de: stopping tablet replica
I20260619 09:01:23.158650 5525 raft_consensus.cc:2229] T 6651201b8e5b4963842ce583664ea750 P 97fd1f60950a4384abccd959deed07de [term 1 LEADER]: Raft consensus shutting down.
I20260619 09:01:23.158728 5525 raft_consensus.cc:2258] T 6651201b8e5b4963842ce583664ea750 P 97fd1f60950a4384abccd959deed07de [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 09:01:23.159176 5525 ts_tablet_manager.cc:1905] T 6651201b8e5b4963842ce583664ea750 P 97fd1f60950a4384abccd959deed07de: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 09:01:23.160758 5525 ts_tablet_manager.cc:1918] T 6651201b8e5b4963842ce583664ea750 P 97fd1f60950a4384abccd959deed07de: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 09:01:23.160830 5525 log.cc:1196] T 6651201b8e5b4963842ce583664ea750 P 97fd1f60950a4384abccd959deed07de: Deleting WAL directory at /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781859658134510-392-0/minicluster-data/ts-2/wal/wals/6651201b8e5b4963842ce583664ea750
I20260619 09:01:23.161013 5524 ts_tablet_manager.cc:1905] T 6651201b8e5b4963842ce583664ea750 P 58447d2f949d44ff9293003bc624af4a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 09:01:23.161085 5525 ts_tablet_manager.cc:1939] T 6651201b8e5b4963842ce583664ea750 P 97fd1f60950a4384abccd959deed07de: Deleting consensus metadata
I20260619 09:01:23.161459 5022 catalog_manager.cc:4841] TS 97fd1f60950a4384abccd959deed07de (127.0.98.3:40775): tablet 6651201b8e5b4963842ce583664ea750 (table test-table [id=568471bb209640b7be9a0ce27b5c6ca8]) successfully deleted
I20260619 09:01:23.162671 5524 ts_tablet_manager.cc:1918] T 6651201b8e5b4963842ce583664ea750 P 58447d2f949d44ff9293003bc624af4a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 09:01:23.162839 5524 log.cc:1196] T 6651201b8e5b4963842ce583664ea750 P 58447d2f949d44ff9293003bc624af4a: Deleting WAL directory at /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781859658134510-392-0/minicluster-data/ts-1/wal/wals/6651201b8e5b4963842ce583664ea750
I20260619 09:01:23.163146 5524 ts_tablet_manager.cc:1939] T 6651201b8e5b4963842ce583664ea750 P 58447d2f949d44ff9293003bc624af4a: Deleting consensus metadata
I20260619 09:01:23.163583 5021 catalog_manager.cc:4841] TS 58447d2f949d44ff9293003bc624af4a (127.0.98.2:39999): tablet 6651201b8e5b4963842ce583664ea750 (table test-table [id=568471bb209640b7be9a0ce27b5c6ca8]) successfully deleted
W20260619 09:01:23.164166 5022 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:40145 (error 108)
I20260619 09:01:23.164654 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ez5NB/build/release/bin/kudu with pid 5228
I20260619 09:01:23.164662 5498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:23.164535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40145 (local address 127.0.0.1:53994)
0619 09:01:23.164586 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0619 09:01:23.164628 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40145: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260619 09:01:23.165370 5022 catalog_manager.cc:4584] TS 8f3b3dfb3b1740a98126ff1656b767da (127.0.98.1:40145): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6651201b8e5b4963842ce583664ea750: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40145: connect: Connection refused (error 111)
I20260619 09:01:23.170095 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ez5NB/build/release/bin/kudu with pid 5359
I20260619 09:01:23.173774 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ez5NB/build/release/bin/kudu with pid 5006
2026-06-19T09:01:23Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (659 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260619 09:01:23.191715 392 mini_oidc.cc:152] Starting JWKS server
I20260619 09:01:23.192682 392 webserver.cc:460] Webserver started at https://127.0.0.1:43105/ using document root <none> and password file <none>
I20260619 09:01:23.192756 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260619 09:01:23.192880 392 webserver.cc:460] Webserver started at http://127.0.0.1:39879/ using document root <none> and password file <none>
2026-06-19T09:01:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T09:01:23Z Disabled control of system clock
I20260619 09:01:23.199590 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ez5NB/build/release/bin/kudu
/tmp/dist-test-task4ez5NB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781859658134510-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781859658134510-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781859658134510-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781859658134510-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39695
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:40991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39695
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43105/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781859658134510-392-0/testchainca.cert with env {}
W20260619 09:01:23.275858 5533 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 09:01:23.276043 5533 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 09:01:23.276063 5533 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 09:01:23.277349 5533 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 09:01:23.277400 5533 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 09:01:23.277411 5533 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 09:01:23.277422 5533 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 09:01:23.277433 5533 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260619 09:01:23.277444 5533 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260619 09:01:23.277453 5533 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:43105/jwks.json
I20260619 09:01:23.278950 5533 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781859658134510-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781859658134510-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39695
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39695
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43105/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781859658134510-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781859658134510-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5533
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781859658134510-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 09:01:23.279142 5533 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 09:01:23.279295 5533 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 09:01:23.283852 5539 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:01:23.283855 5541 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:01:23.283870 5533 server_base.cc:1034] running on GCE node
W20260619 09:01:23.283878 5538 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:01:23.284245 5533 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 09:01:23.284512 5533 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 09:01:23.285650 5533 hybrid_clock.cc:648] HybridClock initialized: now 1781859683285633 us; error 33 us; skew 500 ppm
I20260619 09:01:23.286767 5533 webserver.cc:460] Webserver started at http://127.0.98.62:46817/ using document root <none> and password file <none>
I20260619 09:01:23.287029 5533 fs_manager.cc:359] Metadata directory not provided
I20260619 09:01:23.287081 5533 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 09:01:23.287195 5533 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 09:01:23.288166 5533 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781859658134510-392-0/minicluster-data/master-0/data/instance:
uuid: "0e406d68a70b437b94f391a30fdb841f"
format_stamp: "Formatted at 2026-06-19 09:01:23 on dist-test-slave-mtdr"
server_key: "eec1a9d437464312e3daeec7c1333316"
server_key_iv: "6f2c70372a368670739c8bf38e95b428"
server_key_version: "encryptionkey@0"
I20260619 09:01:23.288484 5533 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781859658134510-392-0/minicluster-data/master-0/wal/instance:
uuid: "0e406d68a70b437b94f391a30fdb841f"
format_stamp: "Formatted at 2026-06-19 09:01:23 on dist-test-slave-mtdr"
server_key: "eec1a9d437464312e3daeec7c1333316"
server_key_iv: "6f2c70372a368670739c8bf38e95b428"
server_key_version: "encryptionkey@0"
I20260619 09:01:23.289669 5533 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 09:01:23.290328 5547 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 09:01:23.290504 5533 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 09:01:23.290584 5533 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781859658134510-392-0/minicluster-data/master-0/data,/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781859658134510-392-0/minicluster-data/master-0/wal
uuid: "0e406d68a70b437b94f391a30fdb841f"
format_stamp: "Formatted at 2026-06-19 09:01:23 on dist-test-slave-mtdr"
server_key: "eec1a9d437464312e3daeec7c1333316"
server_key_iv: "6f2c70372a368670739c8bf38e95b428"
server_key_version: "encryptionkey@0"
I20260619 09:01:23.290643 5533 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781859658134510-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781859658134510-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781859658134510-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 09:01:23.331961 5533 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 09:01:23.332258 5533 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 09:01:23.332391 5533 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 09:01:23.335904 5533 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39695
I20260619 09:01:23.335942 5599 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39695 every 8 connection(s)
I20260619 09:01:23.336243 5533 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781859658134510-392-0/minicluster-data/master-0/data/info.pb
I20260619 09:01:23.337146 5600 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 09:01:23.339329 5600 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0e406d68a70b437b94f391a30fdb841f: Bootstrap starting.
I20260619 09:01:23.339830 5600 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0e406d68a70b437b94f391a30fdb841f: Neither blocks nor log segments found. Creating new log.
I20260619 09:01:23.340054 5600 log.cc:824] T 00000000000000000000000000000000 P 0e406d68a70b437b94f391a30fdb841f: Log is configured to *not* fsync() on all Append() calls
I20260619 09:01:23.340644 5600 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0e406d68a70b437b94f391a30fdb841f: No bootstrap required, opened a new log
I20260619 09:01:23.342346 5600 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0e406d68a70b437b94f391a30fdb841f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e406d68a70b437b94f391a30fdb841f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39695 } }
I20260619 09:01:23.342510 5600 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0e406d68a70b437b94f391a30fdb841f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 09:01:23.342541 5600 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0e406d68a70b437b94f391a30fdb841f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e406d68a70b437b94f391a30fdb841f, State: Initialized, Role: FOLLOWER
I20260619 09:01:23.342669 5600 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0e406d68a70b437b94f391a30fdb841f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e406d68a70b437b94f391a30fdb841f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39695 } }
I20260619 09:01:23.342722 5600 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0e406d68a70b437b94f391a30fdb841f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 09:01:23.342746 5600 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0e406d68a70b437b94f391a30fdb841f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 09:01:23.342774 5600 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0e406d68a70b437b94f391a30fdb841f [term 0 FOLLOWER]: Advancing to term 1
I20260619 09:01:23.343243 5600 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0e406d68a70b437b94f391a30fdb841f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e406d68a70b437b94f391a30fdb841f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39695 } }
I20260619 09:01:23.343323 5600 leader_election.cc:302] T 00000000000000000000000000000000 P 0e406d68a70b437b94f391a30fdb841f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0e406d68a70b437b94f391a30fdb841f; no voters:
I20260619 09:01:23.343454 5600 leader_election.cc:288] T 00000000000000000000000000000000 P 0e406d68a70b437b94f391a30fdb841f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 09:01:23.343495 5603 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0e406d68a70b437b94f391a30fdb841f [term 1 FOLLOWER]: Leader election won for term 1
I20260619 09:01:23.343575 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ez5NB/build/release/bin/kudu as pid 5533
I20260619 09:01:23.343659 5603 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0e406d68a70b437b94f391a30fdb841f [term 1 LEADER]: Becoming Leader. State: Replica: 0e406d68a70b437b94f391a30fdb841f, State: Running, Role: LEADER
I20260619 09:01:23.343676 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781859658134510-392-0/minicluster-data/master-0/wal/instance
I20260619 09:01:23.343696 5600 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0e406d68a70b437b94f391a30fdb841f [sys.catalog]: configured and running, proceeding with master startup.
I20260619 09:01:23.343755 5603 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0e406d68a70b437b94f391a30fdb841f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e406d68a70b437b94f391a30fdb841f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39695 } }
I20260619 09:01:23.343843 392 external_mini_cluster.cc:1428] Setting key c4eb83fe1d6c6938c9f0c4edeb19193c
I20260619 09:01:23.344949 5604 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e406d68a70b437b94f391a30fdb841f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0e406d68a70b437b94f391a30fdb841f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e406d68a70b437b94f391a30fdb841f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39695 } } }
I20260619 09:01:23.345026 5604 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e406d68a70b437b94f391a30fdb841f [sys.catalog]: This master's current role is: LEADER
I20260619 09:01:23.345121 5605 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e406d68a70b437b94f391a30fdb841f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0e406d68a70b437b94f391a30fdb841f. Latest consensus state: current_term: 1 leader_uuid: "0e406d68a70b437b94f391a30fdb841f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e406d68a70b437b94f391a30fdb841f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39695 } } }
I20260619 09:01:23.345417 5605 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e406d68a70b437b94f391a30fdb841f [sys.catalog]: This master's current role is: LEADER
W20260619 09:01:23.345831 5619 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0e406d68a70b437b94f391a30fdb841f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 09:01:23.345886 5619 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 09:01:23.345954 5620 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 09:01:23.346098 5620 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 09:01:23.347510 5620 catalog_manager.cc:1347] Generated new cluster ID: aedda7e485f14f0c92e70bfbb465b14d
I20260619 09:01:23.347560 5620 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 09:01:23.350004 5612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:23.344283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54642 (local address 127.0.98.62:39695)
0619 09:01:23.344626 (+ 343us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:23.344635 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:23.344666 (+ 31us) server_negotiation.cc:407] Connection header received
0619 09:01:23.344786 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:23.344795 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:23.344871 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:23.345024 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:01:23.345686 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:23.346980 (+ 1294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:23.348013 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:23.348618 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:23.348721 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:23.349357 (+ 636us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:01:23.349361 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:01:23.349361 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 09:01:23.349549 (+ 188us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:01:23.349587 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 09:01:23.349599 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:01:23.349673 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0619 09:01:23.349746 (+ 73us) server_negotiation.cc:299] Negotiation successful
0619 09:01:23.349815 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":261,"thread_start_us":203,"threads_started":1}
I20260619 09:01:23.350744 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 09:01:23.359522 5620 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 09:01:23.359978 5620 catalog_manager.cc:1504] Loading token signing keys...
I20260619 09:01:23.378173 5620 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0e406d68a70b437b94f391a30fdb841f: Generated new TSK 0
I20260619 09:01:23.378346 5620 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260619 09:01:23.408922 5630 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260619 09:01:23.409030 5630 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
W20260619 09:01:23.409514 5629 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 09:01:23.387056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39695 (local address 127.0.0.1:54644)
0619 09:01:23.387224 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0619 09:01:23.387236 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 09:01:23.387312 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:01:23.387488 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:01:23.387488 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:01:23.387495 (+ 7us) client_negotiation.cc:189] Negotiated authn=JWT
0619 09:01:23.387573 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:23.387575 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:23.388217 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:23.388218 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:23.389127 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:23.389130 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:23.389480 (+ 350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:23.389483 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:23.389552 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:23.389574 (+ 22us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0619 09:01:23.409159 (+ 19585us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43105/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0619 09:01:23.409261 (+ 102us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:39695: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43105/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":68,"threads_started":1}
I20260619 09:01:23.409794 5612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:23.387133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54644 (local address 127.0.98.62:39695)
0619 09:01:23.387182 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:23.387184 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:23.387287 (+ 103us) server_negotiation.cc:407] Connection header received
0619 09:01:23.387407 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:23.387408 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:23.387417 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:23.387472 (+ 55us) server_negotiation.cc:226] Negotiated authn=JWT
0619 09:01:23.387637 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:23.388138 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:23.389220 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:23.389413 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:23.389469 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:23.389684 (+ 215us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0619 09:01:23.409005 (+ 19321us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43105/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0619 09:01:23.409734 (+ 729us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:54644: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43105/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":27}
W20260619 09:01:23.409842 5612 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:54644: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43105/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.0.98.62:39695: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43105/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260619 09:01:23.410122 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ez5NB/build/release/bin/kudu with pid 5533
2026-06-19T09:01:23Z chronyd exiting
I20260619 09:01:23.445132 392 test_util.cc:175] -----------------------------------------------
I20260619 09:01:23.445214 392 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781859658134510-392-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (254 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260619 09:01:23.445907 392 mini_oidc.cc:152] Starting JWKS server
I20260619 09:01:23.446555 392 webserver.cc:460] Webserver started at https://127.0.0.1:39373/ using document root <none> and password file <none>
I20260619 09:01:23.446592 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260619 09:01:23.446702 392 webserver.cc:460] Webserver started at http://127.0.0.1:36737/ using document root <none> and password file <none>
2026-06-19T09:01:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T09:01:23Z Disabled control of system clock
I20260619 09:01:23.453313 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ez5NB/build/release/bin/kudu
/tmp/dist-test-task4ez5NB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781859658134510-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781859658134510-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781859658134510-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781859658134510-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43133
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:32909
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43133
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39373/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781859658134510-392-0/testchainca.cert with env {}
W20260619 09:01:23.529269 5636 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 09:01:23.529460 5636 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 09:01:23.529488 5636 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 09:01:23.530632 5636 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 09:01:23.530670 5636 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 09:01:23.530686 5636 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 09:01:23.530702 5636 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 09:01:23.530716 5636 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260619 09:01:23.530733 5636 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260619 09:01:23.530751 5636 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:39373/jwks.json
I20260619 09:01:23.532218 5636 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32909
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781859658134510-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781859658134510-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43133
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43133
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39373/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781859658134510-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781859658134510-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5636
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781859658134510-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 09:01:23.532444 5636 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 09:01:23.532627 5636 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 09:01:23.537174 5644 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:01:23.537165 5642 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:01:23.537235 5636 server_base.cc:1034] running on GCE node
W20260619 09:01:23.537174 5641 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:01:23.537616 5636 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 09:01:23.537858 5636 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 09:01:23.538995 5636 hybrid_clock.cc:648] HybridClock initialized: now 1781859683538960 us; error 48 us; skew 500 ppm
I20260619 09:01:23.540104 5636 webserver.cc:460] Webserver started at http://127.0.98.62:44259/ using document root <none> and password file <none>
I20260619 09:01:23.540336 5636 fs_manager.cc:359] Metadata directory not provided
I20260619 09:01:23.540383 5636 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 09:01:23.540499 5636 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 09:01:23.541455 5636 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781859658134510-392-0/minicluster-data/master-0/data/instance:
uuid: "741a18103aef4d2582134db9c55cfde7"
format_stamp: "Formatted at 2026-06-19 09:01:23 on dist-test-slave-mtdr"
server_key: "fb113b6271bb61613fcdb7c89173b1e4"
server_key_iv: "daf5b59ece2702f2f51efeac70db827a"
server_key_version: "encryptionkey@0"
I20260619 09:01:23.541759 5636 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781859658134510-392-0/minicluster-data/master-0/wal/instance:
uuid: "741a18103aef4d2582134db9c55cfde7"
format_stamp: "Formatted at 2026-06-19 09:01:23 on dist-test-slave-mtdr"
server_key: "fb113b6271bb61613fcdb7c89173b1e4"
server_key_iv: "daf5b59ece2702f2f51efeac70db827a"
server_key_version: "encryptionkey@0"
I20260619 09:01:23.542937 5636 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 09:01:23.543567 5650 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 09:01:23.543712 5636 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 09:01:23.543789 5636 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781859658134510-392-0/minicluster-data/master-0/data,/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781859658134510-392-0/minicluster-data/master-0/wal
uuid: "741a18103aef4d2582134db9c55cfde7"
format_stamp: "Formatted at 2026-06-19 09:01:23 on dist-test-slave-mtdr"
server_key: "fb113b6271bb61613fcdb7c89173b1e4"
server_key_iv: "daf5b59ece2702f2f51efeac70db827a"
server_key_version: "encryptionkey@0"
I20260619 09:01:23.543851 5636 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781859658134510-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781859658134510-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781859658134510-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 09:01:23.557964 5636 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 09:01:23.558226 5636 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 09:01:23.558341 5636 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 09:01:23.561725 5636 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43133
I20260619 09:01:23.561762 5702 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43133 every 8 connection(s)
I20260619 09:01:23.562043 5636 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781859658134510-392-0/minicluster-data/master-0/data/info.pb
I20260619 09:01:23.562906 5703 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 09:01:23.564953 5703 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 741a18103aef4d2582134db9c55cfde7: Bootstrap starting.
I20260619 09:01:23.565438 5703 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 741a18103aef4d2582134db9c55cfde7: Neither blocks nor log segments found. Creating new log.
I20260619 09:01:23.565645 5703 log.cc:824] T 00000000000000000000000000000000 P 741a18103aef4d2582134db9c55cfde7: Log is configured to *not* fsync() on all Append() calls
I20260619 09:01:23.566182 5703 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 741a18103aef4d2582134db9c55cfde7: No bootstrap required, opened a new log
I20260619 09:01:23.567168 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ez5NB/build/release/bin/kudu as pid 5636
I20260619 09:01:23.567251 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781859658134510-392-0/minicluster-data/master-0/wal/instance
I20260619 09:01:23.567399 392 external_mini_cluster.cc:1428] Setting key d13b11485b914b4b15e79de2bb599bce
I20260619 09:01:23.567836 5703 raft_consensus.cc:348] T 00000000000000000000000000000000 P 741a18103aef4d2582134db9c55cfde7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "741a18103aef4d2582134db9c55cfde7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43133 } }
I20260619 09:01:23.567997 5703 raft_consensus.cc:374] T 00000000000000000000000000000000 P 741a18103aef4d2582134db9c55cfde7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 09:01:23.568041 5703 raft_consensus.cc:729] T 00000000000000000000000000000000 P 741a18103aef4d2582134db9c55cfde7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 741a18103aef4d2582134db9c55cfde7, State: Initialized, Role: FOLLOWER
I20260619 09:01:23.568156 5703 consensus_queue.cc:260] T 00000000000000000000000000000000 P 741a18103aef4d2582134db9c55cfde7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "741a18103aef4d2582134db9c55cfde7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43133 } }
I20260619 09:01:23.568235 5703 raft_consensus.cc:388] T 00000000000000000000000000000000 P 741a18103aef4d2582134db9c55cfde7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 09:01:23.568269 5703 raft_consensus.cc:482] T 00000000000000000000000000000000 P 741a18103aef4d2582134db9c55cfde7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 09:01:23.568313 5703 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 741a18103aef4d2582134db9c55cfde7 [term 0 FOLLOWER]: Advancing to term 1
I20260619 09:01:23.568923 5703 raft_consensus.cc:504] T 00000000000000000000000000000000 P 741a18103aef4d2582134db9c55cfde7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "741a18103aef4d2582134db9c55cfde7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43133 } }
I20260619 09:01:23.569021 5703 leader_election.cc:302] T 00000000000000000000000000000000 P 741a18103aef4d2582134db9c55cfde7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 741a18103aef4d2582134db9c55cfde7; no voters:
I20260619 09:01:23.569154 5703 leader_election.cc:288] T 00000000000000000000000000000000 P 741a18103aef4d2582134db9c55cfde7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 09:01:23.569203 5708 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 741a18103aef4d2582134db9c55cfde7 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 09:01:23.569341 5708 raft_consensus.cc:686] T 00000000000000000000000000000000 P 741a18103aef4d2582134db9c55cfde7 [term 1 LEADER]: Becoming Leader. State: Replica: 741a18103aef4d2582134db9c55cfde7, State: Running, Role: LEADER
I20260619 09:01:23.569388 5703 sys_catalog.cc:564] T 00000000000000000000000000000000 P 741a18103aef4d2582134db9c55cfde7 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 09:01:23.569469 5708 consensus_queue.cc:237] T 00000000000000000000000000000000 P 741a18103aef4d2582134db9c55cfde7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "741a18103aef4d2582134db9c55cfde7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43133 } }
I20260619 09:01:23.569816 5708 sys_catalog.cc:455] T 00000000000000000000000000000000 P 741a18103aef4d2582134db9c55cfde7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 741a18103aef4d2582134db9c55cfde7. Latest consensus state: current_term: 1 leader_uuid: "741a18103aef4d2582134db9c55cfde7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "741a18103aef4d2582134db9c55cfde7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43133 } } }
I20260619 09:01:23.569824 5710 sys_catalog.cc:455] T 00000000000000000000000000000000 P 741a18103aef4d2582134db9c55cfde7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "741a18103aef4d2582134db9c55cfde7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "741a18103aef4d2582134db9c55cfde7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43133 } } }
I20260619 09:01:23.569924 5708 sys_catalog.cc:458] T 00000000000000000000000000000000 P 741a18103aef4d2582134db9c55cfde7 [sys.catalog]: This master's current role is: LEADER
I20260619 09:01:23.569945 5710 sys_catalog.cc:458] T 00000000000000000000000000000000 P 741a18103aef4d2582134db9c55cfde7 [sys.catalog]: This master's current role is: LEADER
I20260619 09:01:23.570464 5717 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 09:01:23.570995 5717 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 09:01:23.572350 5717 catalog_manager.cc:1347] Generated new cluster ID: 7ea7226df0074a6b9b2ecf179885ca54
I20260619 09:01:23.572394 5717 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 09:01:23.573365 5707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:23.567821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57080 (local address 127.0.98.62:43133)
0619 09:01:23.568024 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:23.568032 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:23.568067 (+ 35us) server_negotiation.cc:407] Connection header received
0619 09:01:23.568172 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:23.568179 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:23.568246 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:23.568384 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:01:23.568554 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:23.570253 (+ 1699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:23.571325 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:23.571920 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:23.572030 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:23.572676 (+ 646us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:01:23.572679 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:01:23.572680 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 09:01:23.572877 (+ 197us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:01:23.572917 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 09:01:23.572932 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:01:23.572998 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0619 09:01:23.573060 (+ 62us) server_negotiation.cc:299] Negotiation successful
0619 09:01:23.573136 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":85,"threads_started":1}
I20260619 09:01:23.574169 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 09:01:23.582513 5717 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 09:01:23.583012 5717 catalog_manager.cc:1504] Loading token signing keys...
I20260619 09:01:23.586282 5717 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 741a18103aef4d2582134db9c55cfde7: Generated new TSK 0
I20260619 09:01:23.586426 5717 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260619 09:01:23.616359 5733 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260619 09:01:23.616436 5733 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260619 09:01:23.616686 5707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:23.595247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57088 (local address 127.0.98.62:43133)
0619 09:01:23.595290 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:23.595292 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:23.595375 (+ 83us) server_negotiation.cc:407] Connection header received
0619 09:01:23.595444 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:23.595445 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:23.595457 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:23.595511 (+ 54us) server_negotiation.cc:226] Negotiated authn=JWT
0619 09:01:23.595622 (+ 111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:23.596090 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:23.596807 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:23.597006 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:23.597057 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:23.597204 (+ 147us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0619 09:01:23.616479 (+ 19275us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39373/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0619 09:01:23.616622 (+ 143us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:57088: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39373/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":25}
W20260619 09:01:23.616750 5707 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:57088: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39373/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260619 09:01:23.616871 5732 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 09:01:23.595192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43133 (local address 127.0.0.1:57088)
0619 09:01:23.595317 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0619 09:01:23.595326 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 09:01:23.595381 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:01:23.595514 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:01:23.595514 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:01:23.595516 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0619 09:01:23.595567 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:23.595569 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:23.596157 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:23.596158 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:23.596734 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:23.596736 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:23.597061 (+ 325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:23.597065 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:23.597122 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:23.597126 (+ 4us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0619 09:01:23.616603 (+ 19477us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39373/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0619 09:01:23.616711 (+ 108us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:43133: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39373/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":72,"threads_started":1}
I20260619 09:01:23.617101 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ez5NB/build/release/bin/kudu with pid 5636
2026-06-19T09:01:23Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (207 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260619 09:01:23.653522 392 mini_oidc.cc:152] Starting JWKS server
I20260619 09:01:23.654222 392 webserver.cc:460] Webserver started at https://127.0.0.1:33105/ using document root <none> and password file <none>
I20260619 09:01:23.654268 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260619 09:01:23.654423 392 webserver.cc:460] Webserver started at http://127.0.0.1:40997/ using document root <none> and password file <none>
2026-06-19T09:01:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T09:01:23Z Disabled control of system clock
I20260619 09:01:23.661345 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ez5NB/build/release/bin/kudu
/tmp/dist-test-task4ez5NB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781859658134510-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781859658134510-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781859658134510-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781859658134510-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39153
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:42891
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39153
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33105/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260619 09:01:23.738567 5739 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 09:01:23.738760 5739 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 09:01:23.738786 5739 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 09:01:23.739956 5739 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 09:01:23.739995 5739 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 09:01:23.740011 5739 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 09:01:23.740028 5739 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 09:01:23.740042 5739 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260619 09:01:23.740059 5739 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260619 09:01:23.740082 5739 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:33105/jwks.json
I20260619 09:01:23.741441 5739 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42891
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781859658134510-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781859658134510-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39153
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39153
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33105/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781859658134510-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5739
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781859658134510-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 09:01:23.741662 5739 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 09:01:23.741925 5739 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 09:01:23.746407 5744 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:01:23.746538 5739 server_base.cc:1034] running on GCE node
W20260619 09:01:23.746407 5745 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:01:23.746417 5747 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:01:23.746898 5739 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 09:01:23.747171 5739 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 09:01:23.748334 5739 hybrid_clock.cc:648] HybridClock initialized: now 1781859683748325 us; error 34 us; skew 500 ppm
I20260619 09:01:23.749509 5739 webserver.cc:460] Webserver started at http://127.0.98.62:37753/ using document root <none> and password file <none>
I20260619 09:01:23.749782 5739 fs_manager.cc:359] Metadata directory not provided
I20260619 09:01:23.749832 5739 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 09:01:23.749967 5739 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 09:01:23.751025 5739 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781859658134510-392-0/minicluster-data/master-0/data/instance:
uuid: "79251db47b144c2aa7eb283769eadedb"
format_stamp: "Formatted at 2026-06-19 09:01:23 on dist-test-slave-mtdr"
server_key: "69b6f2735b0221b94d40fc70a0734c48"
server_key_iv: "2f8541e5d89ab8b32d4f3a746cb59643"
server_key_version: "encryptionkey@0"
I20260619 09:01:23.751353 5739 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781859658134510-392-0/minicluster-data/master-0/wal/instance:
uuid: "79251db47b144c2aa7eb283769eadedb"
format_stamp: "Formatted at 2026-06-19 09:01:23 on dist-test-slave-mtdr"
server_key: "69b6f2735b0221b94d40fc70a0734c48"
server_key_iv: "2f8541e5d89ab8b32d4f3a746cb59643"
server_key_version: "encryptionkey@0"
I20260619 09:01:23.752503 5739 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 09:01:23.753145 5753 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 09:01:23.753307 5739 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260619 09:01:23.753392 5739 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781859658134510-392-0/minicluster-data/master-0/data,/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781859658134510-392-0/minicluster-data/master-0/wal
uuid: "79251db47b144c2aa7eb283769eadedb"
format_stamp: "Formatted at 2026-06-19 09:01:23 on dist-test-slave-mtdr"
server_key: "69b6f2735b0221b94d40fc70a0734c48"
server_key_iv: "2f8541e5d89ab8b32d4f3a746cb59643"
server_key_version: "encryptionkey@0"
I20260619 09:01:23.753460 5739 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781859658134510-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781859658134510-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781859658134510-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 09:01:23.773229 5739 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 09:01:23.773504 5739 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 09:01:23.773624 5739 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 09:01:23.777132 5739 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39153
I20260619 09:01:23.777169 5805 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39153 every 8 connection(s)
I20260619 09:01:23.777451 5739 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781859658134510-392-0/minicluster-data/master-0/data/info.pb
I20260619 09:01:23.778079 5806 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 09:01:23.780571 5806 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 79251db47b144c2aa7eb283769eadedb: Bootstrap starting.
I20260619 09:01:23.781121 5806 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 79251db47b144c2aa7eb283769eadedb: Neither blocks nor log segments found. Creating new log.
I20260619 09:01:23.781374 5806 log.cc:824] T 00000000000000000000000000000000 P 79251db47b144c2aa7eb283769eadedb: Log is configured to *not* fsync() on all Append() calls
I20260619 09:01:23.781965 5806 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 79251db47b144c2aa7eb283769eadedb: No bootstrap required, opened a new log
I20260619 09:01:23.783488 5806 raft_consensus.cc:348] T 00000000000000000000000000000000 P 79251db47b144c2aa7eb283769eadedb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79251db47b144c2aa7eb283769eadedb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39153 } }
I20260619 09:01:23.783620 5806 raft_consensus.cc:374] T 00000000000000000000000000000000 P 79251db47b144c2aa7eb283769eadedb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 09:01:23.783643 5806 raft_consensus.cc:729] T 00000000000000000000000000000000 P 79251db47b144c2aa7eb283769eadedb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79251db47b144c2aa7eb283769eadedb, State: Initialized, Role: FOLLOWER
I20260619 09:01:23.783775 5806 consensus_queue.cc:260] T 00000000000000000000000000000000 P 79251db47b144c2aa7eb283769eadedb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79251db47b144c2aa7eb283769eadedb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39153 } }
I20260619 09:01:23.783840 5806 raft_consensus.cc:388] T 00000000000000000000000000000000 P 79251db47b144c2aa7eb283769eadedb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 09:01:23.783864 5806 raft_consensus.cc:482] T 00000000000000000000000000000000 P 79251db47b144c2aa7eb283769eadedb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 09:01:23.783910 5806 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 79251db47b144c2aa7eb283769eadedb [term 0 FOLLOWER]: Advancing to term 1
I20260619 09:01:23.784400 5806 raft_consensus.cc:504] T 00000000000000000000000000000000 P 79251db47b144c2aa7eb283769eadedb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79251db47b144c2aa7eb283769eadedb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39153 } }
I20260619 09:01:23.784482 5806 leader_election.cc:302] T 00000000000000000000000000000000 P 79251db47b144c2aa7eb283769eadedb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 79251db47b144c2aa7eb283769eadedb; no voters:
I20260619 09:01:23.784633 5806 leader_election.cc:288] T 00000000000000000000000000000000 P 79251db47b144c2aa7eb283769eadedb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 09:01:23.784678 5809 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 79251db47b144c2aa7eb283769eadedb [term 1 FOLLOWER]: Leader election won for term 1
I20260619 09:01:23.784806 5809 raft_consensus.cc:686] T 00000000000000000000000000000000 P 79251db47b144c2aa7eb283769eadedb [term 1 LEADER]: Becoming Leader. State: Replica: 79251db47b144c2aa7eb283769eadedb, State: Running, Role: LEADER
I20260619 09:01:23.784890 5806 sys_catalog.cc:564] T 00000000000000000000000000000000 P 79251db47b144c2aa7eb283769eadedb [sys.catalog]: configured and running, proceeding with master startup.
I20260619 09:01:23.784894 5809 consensus_queue.cc:237] T 00000000000000000000000000000000 P 79251db47b144c2aa7eb283769eadedb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79251db47b144c2aa7eb283769eadedb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39153 } }
I20260619 09:01:23.785043 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ez5NB/build/release/bin/kudu as pid 5739
I20260619 09:01:23.785141 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781859658134510-392-0/minicluster-data/master-0/wal/instance
I20260619 09:01:23.785344 392 external_mini_cluster.cc:1428] Setting key 439cd85971280b93676ad65a8a596662
I20260619 09:01:23.785868 5810 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79251db47b144c2aa7eb283769eadedb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "79251db47b144c2aa7eb283769eadedb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79251db47b144c2aa7eb283769eadedb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39153 } } }
I20260619 09:01:23.785966 5810 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79251db47b144c2aa7eb283769eadedb [sys.catalog]: This master's current role is: LEADER
I20260619 09:01:23.785876 5811 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79251db47b144c2aa7eb283769eadedb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 79251db47b144c2aa7eb283769eadedb. Latest consensus state: current_term: 1 leader_uuid: "79251db47b144c2aa7eb283769eadedb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79251db47b144c2aa7eb283769eadedb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39153 } } }
I20260619 09:01:23.786185 5811 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79251db47b144c2aa7eb283769eadedb [sys.catalog]: This master's current role is: LEADER
W20260619 09:01:23.787045 5827 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 79251db47b144c2aa7eb283769eadedb: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 09:01:23.787101 5827 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 09:01:23.787150 5820 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 09:01:23.787323 5820 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 09:01:23.788607 5820 catalog_manager.cc:1347] Generated new cluster ID: a864f33746504ae4ba7be820dfd2d974
I20260619 09:01:23.788662 5820 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 09:01:23.790848 5821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:23.786011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46450 (local address 127.0.98.62:39153)
0619 09:01:23.786224 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:23.786234 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:23.786280 (+ 46us) server_negotiation.cc:407] Connection header received
0619 09:01:23.786416 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:23.786424 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:23.786500 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:23.786635 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:01:23.786838 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:23.787983 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:23.789017 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:23.789424 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:23.789516 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:23.790240 (+ 724us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:01:23.790243 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:01:23.790243 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 09:01:23.790402 (+ 159us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:01:23.790434 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 09:01:23.790445 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:01:23.790496 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0619 09:01:23.790550 (+ 54us) server_negotiation.cc:299] Negotiation successful
0619 09:01:23.790612 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92,"thread_start_us":67,"threads_started":1}
I20260619 09:01:23.791697 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 09:01:23.792963 5820 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 09:01:23.793432 5820 catalog_manager.cc:1504] Loading token signing keys...
I20260619 09:01:23.801200 5820 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 79251db47b144c2aa7eb283769eadedb: Generated new TSK 0
I20260619 09:01:23.801370 5820 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 09:01:23.811282 5821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:23.807558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46464 (local address 127.0.98.62:39153)
0619 09:01:23.807615 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:23.807616 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:23.807730 (+ 114us) server_negotiation.cc:407] Connection header received
0619 09:01:23.807791 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:23.807792 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:23.807803 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:23.807867 (+ 64us) server_negotiation.cc:226] Negotiated authn=JWT
0619 09:01:23.807987 (+ 120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:23.808455 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:23.809485 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:23.809815 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:23.809882 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:23.810124 (+ 242us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0619 09:01:23.811089 (+ 965us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33105/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0619 09:01:23.811208 (+ 119us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:46464: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33105/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":35}
W20260619 09:01:23.811335 5821 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:46464: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33105/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260619 09:01:23.811614 5835 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 09:01:23.807506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39153 (local address 127.0.0.1:46464)
0619 09:01:23.807665 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0619 09:01:23.807674 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 09:01:23.807726 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:01:23.807867 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:01:23.807868 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:01:23.807869 (+ 1us) client_negotiation.cc:189] Negotiated authn=JWT
0619 09:01:23.807932 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:23.807934 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:23.808542 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:23.808543 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:23.809407 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:23.809410 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:23.809914 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:23.809919 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:23.810002 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:23.810009 (+ 7us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0619 09:01:23.811411 (+ 1402us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33105/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0619 09:01:23.811505 (+ 94us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:39153: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33105/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":79,"threads_started":1}
I20260619 09:01:23.815979 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ez5NB/build/release/bin/kudu with pid 5739
E20260619 09:01:23.823438 5836 webserver.cc:533] Webserver: sslize failed:
E20260619 09:01:23.823504 5836 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-06-19T09:01:23Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (199 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-06-19T09:01:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T09:01:23Z Disabled control of system clock
I20260619 09:01:23.859066 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ez5NB/build/release/bin/kudu
/tmp/dist-test-task4ez5NB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781859658134510-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781859658134510-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781859658134510-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781859658134510-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34019
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:36387
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34019
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781859658134510-392-0/testchainca.cert with env {}
W20260619 09:01:23.934262 5840 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 09:01:23.934450 5840 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 09:01:23.934479 5840 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 09:01:23.935622 5840 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 09:01:23.935664 5840 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 09:01:23.935680 5840 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 09:01:23.935696 5840 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 09:01:23.935711 5840 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260619 09:01:23.935729 5840 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260619 09:01:23.935746 5840 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260619 09:01:23.937175 5840 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36387
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781859658134510-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781859658134510-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34019
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34019
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781859658134510-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781859658134510-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5840
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781859658134510-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 09:01:23.937403 5840 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 09:01:23.937583 5840 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 09:01:23.941947 5848 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:01:23.941951 5845 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:01:23.941962 5846 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:01:23.941967 5840 server_base.cc:1034] running on GCE node
I20260619 09:01:23.942327 5840 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 09:01:23.942564 5840 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 09:01:23.943687 5840 hybrid_clock.cc:648] HybridClock initialized: now 1781859683943676 us; error 26 us; skew 500 ppm
I20260619 09:01:23.944743 5840 webserver.cc:460] Webserver started at http://127.0.98.62:36165/ using document root <none> and password file <none>
I20260619 09:01:23.944947 5840 fs_manager.cc:359] Metadata directory not provided
I20260619 09:01:23.944990 5840 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 09:01:23.945106 5840 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 09:01:23.946063 5840 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781859658134510-392-0/minicluster-data/master-0/data/instance:
uuid: "4a29f0b755f24d9e817461ed3ae11bc9"
format_stamp: "Formatted at 2026-06-19 09:01:23 on dist-test-slave-mtdr"
server_key: "8d480f6cfc21fd43a360933d2962d5f7"
server_key_iv: "f42ef3b50950c6cee0764a104f10bc91"
server_key_version: "encryptionkey@0"
I20260619 09:01:23.946367 5840 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781859658134510-392-0/minicluster-data/master-0/wal/instance:
uuid: "4a29f0b755f24d9e817461ed3ae11bc9"
format_stamp: "Formatted at 2026-06-19 09:01:23 on dist-test-slave-mtdr"
server_key: "8d480f6cfc21fd43a360933d2962d5f7"
server_key_iv: "f42ef3b50950c6cee0764a104f10bc91"
server_key_version: "encryptionkey@0"
I20260619 09:01:23.947597 5840 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 09:01:23.948244 5854 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 09:01:23.948385 5840 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 09:01:23.948459 5840 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781859658134510-392-0/minicluster-data/master-0/data,/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781859658134510-392-0/minicluster-data/master-0/wal
uuid: "4a29f0b755f24d9e817461ed3ae11bc9"
format_stamp: "Formatted at 2026-06-19 09:01:23 on dist-test-slave-mtdr"
server_key: "8d480f6cfc21fd43a360933d2962d5f7"
server_key_iv: "f42ef3b50950c6cee0764a104f10bc91"
server_key_version: "encryptionkey@0"
I20260619 09:01:23.948518 5840 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781859658134510-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781859658134510-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781859658134510-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 09:01:23.971779 5840 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 09:01:23.972059 5840 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 09:01:23.972179 5840 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 09:01:23.975790 5840 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34019
I20260619 09:01:23.975898 5906 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34019 every 8 connection(s)
I20260619 09:01:23.976195 5840 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781859658134510-392-0/minicluster-data/master-0/data/info.pb
I20260619 09:01:23.977159 5907 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 09:01:23.979291 5907 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a29f0b755f24d9e817461ed3ae11bc9: Bootstrap starting.
I20260619 09:01:23.979780 5907 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4a29f0b755f24d9e817461ed3ae11bc9: Neither blocks nor log segments found. Creating new log.
I20260619 09:01:23.980005 5907 log.cc:824] T 00000000000000000000000000000000 P 4a29f0b755f24d9e817461ed3ae11bc9: Log is configured to *not* fsync() on all Append() calls
I20260619 09:01:23.980619 5907 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a29f0b755f24d9e817461ed3ae11bc9: No bootstrap required, opened a new log
I20260619 09:01:23.982116 5907 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4a29f0b755f24d9e817461ed3ae11bc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a29f0b755f24d9e817461ed3ae11bc9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34019 } }
I20260619 09:01:23.982244 5907 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4a29f0b755f24d9e817461ed3ae11bc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 09:01:23.982264 5907 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4a29f0b755f24d9e817461ed3ae11bc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a29f0b755f24d9e817461ed3ae11bc9, State: Initialized, Role: FOLLOWER
I20260619 09:01:23.982374 5907 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4a29f0b755f24d9e817461ed3ae11bc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a29f0b755f24d9e817461ed3ae11bc9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34019 } }
I20260619 09:01:23.982448 5907 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4a29f0b755f24d9e817461ed3ae11bc9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 09:01:23.982481 5907 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4a29f0b755f24d9e817461ed3ae11bc9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 09:01:23.982517 5907 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4a29f0b755f24d9e817461ed3ae11bc9 [term 0 FOLLOWER]: Advancing to term 1
I20260619 09:01:23.982800 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ez5NB/build/release/bin/kudu as pid 5840
I20260619 09:01:23.982915 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781859658134510-392-0/minicluster-data/master-0/wal/instance
I20260619 09:01:23.983098 392 external_mini_cluster.cc:1428] Setting key a7622546d60bd769894ab9170348ffdd
I20260619 09:01:23.983114 5907 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4a29f0b755f24d9e817461ed3ae11bc9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a29f0b755f24d9e817461ed3ae11bc9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34019 } }
I20260619 09:01:23.983208 5907 leader_election.cc:302] T 00000000000000000000000000000000 P 4a29f0b755f24d9e817461ed3ae11bc9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4a29f0b755f24d9e817461ed3ae11bc9; no voters:
I20260619 09:01:23.983353 5907 leader_election.cc:288] T 00000000000000000000000000000000 P 4a29f0b755f24d9e817461ed3ae11bc9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 09:01:23.983455 5910 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4a29f0b755f24d9e817461ed3ae11bc9 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 09:01:23.983587 5907 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4a29f0b755f24d9e817461ed3ae11bc9 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 09:01:23.983605 5910 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4a29f0b755f24d9e817461ed3ae11bc9 [term 1 LEADER]: Becoming Leader. State: Replica: 4a29f0b755f24d9e817461ed3ae11bc9, State: Running, Role: LEADER
I20260619 09:01:23.983767 5910 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4a29f0b755f24d9e817461ed3ae11bc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a29f0b755f24d9e817461ed3ae11bc9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34019 } }
I20260619 09:01:23.984048 5910 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a29f0b755f24d9e817461ed3ae11bc9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4a29f0b755f24d9e817461ed3ae11bc9. Latest consensus state: current_term: 1 leader_uuid: "4a29f0b755f24d9e817461ed3ae11bc9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a29f0b755f24d9e817461ed3ae11bc9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34019 } } }
I20260619 09:01:23.984120 5910 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a29f0b755f24d9e817461ed3ae11bc9 [sys.catalog]: This master's current role is: LEADER
I20260619 09:01:23.984064 5914 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a29f0b755f24d9e817461ed3ae11bc9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4a29f0b755f24d9e817461ed3ae11bc9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a29f0b755f24d9e817461ed3ae11bc9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34019 } } }
I20260619 09:01:23.984263 5914 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a29f0b755f24d9e817461ed3ae11bc9 [sys.catalog]: This master's current role is: LEADER
I20260619 09:01:23.984778 5920 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 09:01:23.985239 5920 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 09:01:23.986776 5920 catalog_manager.cc:1347] Generated new cluster ID: b551846b585e43e58aa318ee391f2765
I20260619 09:01:23.986845 5920 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 09:01:23.988504 5913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:23.983492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58184 (local address 127.0.98.62:34019)
0619 09:01:23.983675 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:23.983683 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:23.983785 (+ 102us) server_negotiation.cc:407] Connection header received
0619 09:01:23.983990 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:23.983997 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:23.984067 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:23.984176 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:01:23.984486 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:23.985648 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:23.986604 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:23.987162 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:23.987277 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:23.987859 (+ 582us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:01:23.987862 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:01:23.987863 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 09:01:23.988030 (+ 167us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:01:23.988060 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 09:01:23.988071 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:01:23.988122 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0619 09:01:23.988194 (+ 72us) server_negotiation.cc:299] Negotiation successful
0619 09:01:23.988342 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":81,"threads_started":1}
I20260619 09:01:23.989264 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 09:01:23.996892 5920 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 09:01:23.997400 5920 catalog_manager.cc:1504] Loading token signing keys...
I20260619 09:01:23.999627 5920 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4a29f0b755f24d9e817461ed3ae11bc9: Generated new TSK 0
I20260619 09:01:23.999775 5920 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 09:01:24.079998 5913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:24.004134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58190 (local address 127.0.98.62:34019)
0619 09:01:24.004174 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:24.004176 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:24.004271 (+ 95us) server_negotiation.cc:407] Connection header received
0619 09:01:24.004367 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:24.004367 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:24.004379 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:24.004436 (+ 57us) server_negotiation.cc:226] Negotiated authn=JWT
0619 09:01:24.004560 (+ 124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:24.005046 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:24.005761 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:24.005970 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:24.006021 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:24.006176 (+ 155us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0619 09:01:24.079746 (+ 73570us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0619 09:01:24.079919 (+ 173us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:58190: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":22}
W20260619 09:01:24.080080 5913 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:58190: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260619 09:01:24.080116 5936 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 09:01:24.004047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34019 (local address 127.0.0.1:58190)
0619 09:01:24.004193 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0619 09:01:24.004208 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 09:01:24.004279 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:01:24.004441 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:01:24.004442 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:01:24.004444 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0619 09:01:24.004498 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:24.004500 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:24.005128 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:24.005129 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:24.005691 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:24.005693 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:24.006035 (+ 342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:24.006038 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:24.006097 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:24.006101 (+ 4us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0619 09:01:24.079884 (+ 73783us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0619 09:01:24.080012 (+ 128us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:34019: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":78,"threads_started":1}
I20260619 09:01:24.080605 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ez5NB/build/release/bin/kudu with pid 5840
2026-06-19T09:01:24Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (244 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (84 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (83 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781859658134510-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 09:01:24 dist-test-slave-mtdr krb5kdc[5948](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 09:01:24 dist-test-slave-mtdr krb5kdc[5948](info): set up 2 sockets
Jun 19 09:01:24 dist-test-slave-mtdr krb5kdc[5948](info): commencing operation
krb5kdc: starting...
W20260619 09:01:25.202586 392 mini_kdc.cc:121] Time spent starting KDC: real 0.938s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781859658134510-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781859658134510-392-0/krb5kdc/test-user.keytab.
Jun 19 09:01:25 dist-test-slave-mtdr krb5kdc[5948](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859685, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T09:01:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T09:01:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781859658134510-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781859658134510-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 09:01:25.311756 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ez5NB/build/release/bin/kudu
/tmp/dist-test-task4ez5NB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781859658134510-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781859658134510-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781859658134510-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781859658134510-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40405
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781859658134510-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39057
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40405
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781859658134510-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781859658134510-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781859658134510-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 09:01:25.387598 5961 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 09:01:25.387782 5961 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 09:01:25.387802 5961 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 09:01:25.388906 5961 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 09:01:25.388944 5961 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 09:01:25.388957 5961 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 09:01:25.388968 5961 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 09:01:25.388981 5961 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 09:01:25.390496 5961 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39057
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781859658134510-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781859658134510-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40405
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781859658134510-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40405
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781859658134510-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5961
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781859658134510-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 09:01:25.390717 5961 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 09:01:25.391008 5961 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 09:01:25.395408 5966 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:01:25.395457 5969 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:01:25.395419 5967 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:01:25.395612 5961 server_base.cc:1034] running on GCE node
I20260619 09:01:25.395833 5961 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 09:01:25.396059 5961 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 09:01:25.397202 5961 hybrid_clock.cc:648] HybridClock initialized: now 1781859685397193 us; error 28 us; skew 500 ppm
Jun 19 09:01:25 dist-test-slave-mtdr krb5kdc[5948](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859685, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 09:01:25.399447 5961 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 09:01:25.399850 5961 webserver.cc:460] Webserver started at http://127.0.98.62:35207/ using document root <none> and password file <none>
I20260619 09:01:25.400048 5961 fs_manager.cc:359] Metadata directory not provided
I20260619 09:01:25.400094 5961 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 09:01:25.400216 5961 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 09:01:25.401162 5961 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781859658134510-392-0/minicluster-data/master-0/data/instance:
uuid: "7053b5ac67a54bf895d0ace0deba7e0d"
format_stamp: "Formatted at 2026-06-19 09:01:25 on dist-test-slave-mtdr"
server_key: "c771a354f5a2008683f1a4e6fc01f2d5"
server_key_iv: "45941e33fae7f83ee0b4a0ff3ae229a5"
server_key_version: "encryptionkey@0"
I20260619 09:01:25.401484 5961 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781859658134510-392-0/minicluster-data/master-0/wal/instance:
uuid: "7053b5ac67a54bf895d0ace0deba7e0d"
format_stamp: "Formatted at 2026-06-19 09:01:25 on dist-test-slave-mtdr"
server_key: "c771a354f5a2008683f1a4e6fc01f2d5"
server_key_iv: "45941e33fae7f83ee0b4a0ff3ae229a5"
server_key_version: "encryptionkey@0"
I20260619 09:01:25.402647 5961 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 09:01:25.403360 5976 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 09:01:25.403507 5961 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260619 09:01:25.403586 5961 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781859658134510-392-0/minicluster-data/master-0/data,/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781859658134510-392-0/minicluster-data/master-0/wal
uuid: "7053b5ac67a54bf895d0ace0deba7e0d"
format_stamp: "Formatted at 2026-06-19 09:01:25 on dist-test-slave-mtdr"
server_key: "c771a354f5a2008683f1a4e6fc01f2d5"
server_key_iv: "45941e33fae7f83ee0b4a0ff3ae229a5"
server_key_version: "encryptionkey@0"
I20260619 09:01:25.403649 5961 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781859658134510-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781859658134510-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781859658134510-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 09:01:25.427202 5961 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 09:01:25.429217 5961 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 09:01:25.429348 5961 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 09:01:25.433022 5961 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40405
I20260619 09:01:25.433066 6028 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40405 every 8 connection(s)
I20260619 09:01:25.433524 5961 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781859658134510-392-0/minicluster-data/master-0/data/info.pb
I20260619 09:01:25.434115 6029 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 09:01:25.435544 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ez5NB/build/release/bin/kudu as pid 5961
I20260619 09:01:25.435637 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781859658134510-392-0/minicluster-data/master-0/wal/instance
I20260619 09:01:25.435798 392 external_mini_cluster.cc:1428] Setting key ed5b897edf882aaca9db8eccd62bd8ff
I20260619 09:01:25.436633 6029 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7053b5ac67a54bf895d0ace0deba7e0d: Bootstrap starting.
I20260619 09:01:25.437199 6029 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7053b5ac67a54bf895d0ace0deba7e0d: Neither blocks nor log segments found. Creating new log.
I20260619 09:01:25.437527 6029 log.cc:824] T 00000000000000000000000000000000 P 7053b5ac67a54bf895d0ace0deba7e0d: Log is configured to *not* fsync() on all Append() calls
I20260619 09:01:25.438208 6029 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7053b5ac67a54bf895d0ace0deba7e0d: No bootstrap required, opened a new log
I20260619 09:01:25.439990 6029 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7053b5ac67a54bf895d0ace0deba7e0d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7053b5ac67a54bf895d0ace0deba7e0d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40405 } }
I20260619 09:01:25.440125 6029 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7053b5ac67a54bf895d0ace0deba7e0d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 09:01:25.440155 6029 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7053b5ac67a54bf895d0ace0deba7e0d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7053b5ac67a54bf895d0ace0deba7e0d, State: Initialized, Role: FOLLOWER
I20260619 09:01:25.440274 6029 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7053b5ac67a54bf895d0ace0deba7e0d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7053b5ac67a54bf895d0ace0deba7e0d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40405 } }
I20260619 09:01:25.440335 6029 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7053b5ac67a54bf895d0ace0deba7e0d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 09:01:25.440387 6029 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7053b5ac67a54bf895d0ace0deba7e0d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 09:01:25.440425 6029 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7053b5ac67a54bf895d0ace0deba7e0d [term 0 FOLLOWER]: Advancing to term 1
I20260619 09:01:25.440907 6029 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7053b5ac67a54bf895d0ace0deba7e0d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7053b5ac67a54bf895d0ace0deba7e0d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40405 } }
I20260619 09:01:25.440990 6029 leader_election.cc:302] T 00000000000000000000000000000000 P 7053b5ac67a54bf895d0ace0deba7e0d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7053b5ac67a54bf895d0ace0deba7e0d; no voters:
I20260619 09:01:25.441140 6029 leader_election.cc:288] T 00000000000000000000000000000000 P 7053b5ac67a54bf895d0ace0deba7e0d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 09:01:25.441174 6034 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7053b5ac67a54bf895d0ace0deba7e0d [term 1 FOLLOWER]: Leader election won for term 1
I20260619 09:01:25.441284 6034 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7053b5ac67a54bf895d0ace0deba7e0d [term 1 LEADER]: Becoming Leader. State: Replica: 7053b5ac67a54bf895d0ace0deba7e0d, State: Running, Role: LEADER
I20260619 09:01:25.441370 6034 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7053b5ac67a54bf895d0ace0deba7e0d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7053b5ac67a54bf895d0ace0deba7e0d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40405 } }
I20260619 09:01:25.441395 6029 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7053b5ac67a54bf895d0ace0deba7e0d [sys.catalog]: configured and running, proceeding with master startup.
Jun 19 09:01:25 dist-test-slave-mtdr krb5kdc[5948](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859685, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 09:01:25.441793 6035 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7053b5ac67a54bf895d0ace0deba7e0d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7053b5ac67a54bf895d0ace0deba7e0d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7053b5ac67a54bf895d0ace0deba7e0d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40405 } } }
I20260619 09:01:25.441805 6036 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7053b5ac67a54bf895d0ace0deba7e0d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7053b5ac67a54bf895d0ace0deba7e0d. Latest consensus state: current_term: 1 leader_uuid: "7053b5ac67a54bf895d0ace0deba7e0d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7053b5ac67a54bf895d0ace0deba7e0d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40405 } } }
I20260619 09:01:25.441890 6035 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7053b5ac67a54bf895d0ace0deba7e0d [sys.catalog]: This master's current role is: LEADER
I20260619 09:01:25.441893 6036 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7053b5ac67a54bf895d0ace0deba7e0d [sys.catalog]: This master's current role is: LEADER
I20260619 09:01:25.442081 6039 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 09:01:25.442523 6039 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 09:01:25.444613 6039 catalog_manager.cc:1347] Generated new cluster ID: 0a7eb7fcc8da4c498e729ca8d939a589
I20260619 09:01:25.444661 6039 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 09:01:25.445693 6032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:25.436349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59292 (local address 127.0.98.62:40405)
0619 09:01:25.436500 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:25.436507 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:25.436531 (+ 24us) server_negotiation.cc:407] Connection header received
0619 09:01:25.436935 (+ 404us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:25.436942 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:25.436990 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:25.437102 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:01:25.437814 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:25.438802 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:25.439529 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:25.439919 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:25.440009 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:25.442374 (+ 2365us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:01:25.442378 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:01:25.442379 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:01:25.442390 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:01:25.444124 (+ 1734us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:01:25.444490 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:01:25.444493 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:01:25.444494 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:01:25.444612 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:01:25.444815 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:01:25.444816 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:01:25.444816 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:01:25.445111 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:01:25.445256 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0619 09:01:25.445350 (+ 94us) server_negotiation.cc:299] Negotiation successful
0619 09:01:25.445486 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":66,"threads_started":1}
I20260619 09:01:25.454129 6039 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 09:01:25.454810 6039 catalog_manager.cc:1504] Loading token signing keys...
I20260619 09:01:25.462960 6039 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7053b5ac67a54bf895d0ace0deba7e0d: Generated new TSK 0
I20260619 09:01:25.463135 6039 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781859658134510-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781859658134510-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 09:01:25.494977 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ez5NB/build/release/bin/kudu
/tmp/dist-test-task4ez5NB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781859658134510-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781859658134510-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781859658134510-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781859658134510-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40405
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781859658134510-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39057
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781859658134510-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781859658134510-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781859658134510-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 09:01:25.593142 6055 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 09:01:25.593338 6055 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 09:01:25.593359 6055 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 09:01:25.594502 6055 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 09:01:25.594542 6055 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 09:01:25.594569 6055 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 09:01:25.596055 6055 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39057
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781859658134510-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781859658134510-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781859658134510-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781859658134510-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40405
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6055
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781859658134510-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 09:01:25.596271 6055 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 09:01:25.596499 6055 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 09:01:25.601193 6061 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:01:25.601227 6060 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:01:25.601347 6063 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:01:25.601399 6055 server_base.cc:1034] running on GCE node
I20260619 09:01:25.601632 6055 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 09:01:25.601845 6055 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 09:01:25.602982 6055 hybrid_clock.cc:648] HybridClock initialized: now 1781859685602961 us; error 28 us; skew 500 ppm
Jun 19 09:01:25 dist-test-slave-mtdr krb5kdc[5948](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859685, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 09:01:25.605345 6055 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 09:01:25.605772 6055 webserver.cc:460] Webserver started at http://127.0.98.1:35349/ using document root <none> and password file <none>
I20260619 09:01:25.605974 6055 fs_manager.cc:359] Metadata directory not provided
I20260619 09:01:25.606029 6055 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 09:01:25.606154 6055 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 09:01:25.607225 6055 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781859658134510-392-0/minicluster-data/ts-0/data/instance:
uuid: "e50bd754e4b94a46b791b0cb34d7fe48"
format_stamp: "Formatted at 2026-06-19 09:01:25 on dist-test-slave-mtdr"
server_key: "f7940b7b569b5ff298dd32cf09242c50"
server_key_iv: "23c40f2451d6f6b1f51456403f3f26bd"
server_key_version: "encryptionkey@0"
I20260619 09:01:25.607548 6055 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781859658134510-392-0/minicluster-data/ts-0/wal/instance:
uuid: "e50bd754e4b94a46b791b0cb34d7fe48"
format_stamp: "Formatted at 2026-06-19 09:01:25 on dist-test-slave-mtdr"
server_key: "f7940b7b569b5ff298dd32cf09242c50"
server_key_iv: "23c40f2451d6f6b1f51456403f3f26bd"
server_key_version: "encryptionkey@0"
I20260619 09:01:25.608724 6055 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 09:01:25.609383 6070 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 09:01:25.609534 6055 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 09:01:25.609611 6055 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781859658134510-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781859658134510-392-0/minicluster-data/ts-0/wal
uuid: "e50bd754e4b94a46b791b0cb34d7fe48"
format_stamp: "Formatted at 2026-06-19 09:01:25 on dist-test-slave-mtdr"
server_key: "f7940b7b569b5ff298dd32cf09242c50"
server_key_iv: "23c40f2451d6f6b1f51456403f3f26bd"
server_key_version: "encryptionkey@0"
I20260619 09:01:25.609670 6055 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781859658134510-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781859658134510-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781859658134510-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 09:01:25.633114 6055 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 09:01:25.635711 6055 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 09:01:25.635850 6055 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 09:01:25.636106 6055 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 09:01:25.636421 6055 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 09:01:25.636469 6055 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 09:01:25.636497 6055 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 09:01:25.636514 6055 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 09:01:25.643062 6055 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46667
I20260619 09:01:25.643131 6183 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46667 every 8 connection(s)
I20260619 09:01:25.643424 6055 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781859658134510-392-0/minicluster-data/ts-0/data/info.pb
Jun 19 09:01:25 dist-test-slave-mtdr krb5kdc[5948](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859685, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 09:01:25.649777 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ez5NB/build/release/bin/kudu as pid 6055
I20260619 09:01:25.649847 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781859658134510-392-0/minicluster-data/ts-0/wal/instance
I20260619 09:01:25.650007 392 external_mini_cluster.cc:1428] Setting key ddbe21517cb175d8b2f718e5230e067a
I20260619 09:01:25.652169 6032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:25.644066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38399 (local address 127.0.98.62:40405)
0619 09:01:25.644107 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:25.644109 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:25.644285 (+ 176us) server_negotiation.cc:407] Connection header received
0619 09:01:25.645180 (+ 895us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:25.645181 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:25.645191 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:25.645243 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:01:25.645795 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:25.646313 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:25.647299 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:25.647526 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:25.647581 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:25.649402 (+ 1821us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:01:25.649403 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:01:25.649404 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:01:25.649410 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:01:25.650805 (+ 1395us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:01:25.651281 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:01:25.651282 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:01:25.651282 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:01:25.651332 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:01:25.651564 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:01:25.651564 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:01:25.651564 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:01:25.651654 (+ 90us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:01:25.651697 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0619 09:01:25.651986 (+ 289us) server_negotiation.cc:299] Negotiation successful
0619 09:01:25.652076 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260619 09:01:25.652343 6186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:25.644041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40405 (local address 127.0.98.1:38399)
0619 09:01:25.644173 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0619 09:01:25.644206 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0619 09:01:25.645082 (+ 876us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:01:25.645299 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:01:25.645303 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:01:25.645536 (+ 233us) client_negotiation.cc:189] Negotiated authn=SASL
0619 09:01:25.645719 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:25.645724 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:25.646379 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:25.646380 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:25.647231 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:25.647233 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:25.647587 (+ 354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:25.647587 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:25.647678 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:25.648090 (+ 412us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 09:01:25.648098 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 09:01:25.649318 (+ 1220us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 09:01:25.650898 (+ 1580us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:01:25.650905 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:01:25.650907 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:01:25.651215 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:01:25.651384 (+ 169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:01:25.651385 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:01:25.651385 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:01:25.651510 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:01:25.651703 (+ 193us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 09:01:25.651707 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 09:01:25.651855 (+ 148us) client_negotiation.cc:769] Sending connection context
0619 09:01:25.651967 (+ 112us) client_negotiation.cc:240] Negotiation successful
0619 09:01:25.652138 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":76,"threads_started":1}
I20260619 09:01:25.652835 6184 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40405
I20260619 09:01:25.652911 6184 heartbeater.cc:464] Registering TS with master...
I20260619 09:01:25.653091 6184 heartbeater.cc:511] Master 127.0.98.62:40405 requested a full tablet report, sending...
I20260619 09:01:25.653582 5993 ts_manager.cc:194] Registered new tserver with Master: e50bd754e4b94a46b791b0cb34d7fe48 (127.0.98.1:46667)
I20260619 09:01:25.654151 5993 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:38399
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781859658134510-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781859658134510-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 09:01:25.672691 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ez5NB/build/release/bin/kudu
/tmp/dist-test-task4ez5NB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781859658134510-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781859658134510-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781859658134510-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781859658134510-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40405
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781859658134510-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39057
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781859658134510-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781859658134510-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781859658134510-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 09:01:25.750671 6189 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 09:01:25.750878 6189 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 09:01:25.750914 6189 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 09:01:25.752027 6189 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 09:01:25.752061 6189 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 09:01:25.752086 6189 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 09:01:25.753469 6189 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39057
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781859658134510-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781859658134510-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781859658134510-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781859658134510-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40405
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6189
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781859658134510-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 09:01:25.753651 6189 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 09:01:25.753822 6189 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 09:01:25.758270 6194 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:01:25.758262 6195 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:01:25.758425 6189 server_base.cc:1034] running on GCE node
W20260619 09:01:25.758275 6197 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:01:25.758673 6189 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 09:01:25.758894 6189 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 09:01:25.760025 6189 hybrid_clock.cc:648] HybridClock initialized: now 1781859685760012 us; error 27 us; skew 500 ppm
Jun 19 09:01:25 dist-test-slave-mtdr krb5kdc[5948](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859685, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 09:01:25.762346 6189 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 09:01:25.762732 6189 webserver.cc:460] Webserver started at http://127.0.98.2:44349/ using document root <none> and password file <none>
I20260619 09:01:25.762935 6189 fs_manager.cc:359] Metadata directory not provided
I20260619 09:01:25.762979 6189 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 09:01:25.763096 6189 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 09:01:25.764014 6189 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781859658134510-392-0/minicluster-data/ts-1/data/instance:
uuid: "a7d0bcde5e2c4bbdbf16a4bafab7ed6a"
format_stamp: "Formatted at 2026-06-19 09:01:25 on dist-test-slave-mtdr"
server_key: "43c1947f89aff0ec13876bc46aee6310"
server_key_iv: "6f3575f03109a98867aa8923523136c6"
server_key_version: "encryptionkey@0"
I20260619 09:01:25.764328 6189 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781859658134510-392-0/minicluster-data/ts-1/wal/instance:
uuid: "a7d0bcde5e2c4bbdbf16a4bafab7ed6a"
format_stamp: "Formatted at 2026-06-19 09:01:25 on dist-test-slave-mtdr"
server_key: "43c1947f89aff0ec13876bc46aee6310"
server_key_iv: "6f3575f03109a98867aa8923523136c6"
server_key_version: "encryptionkey@0"
I20260619 09:01:25.765496 6189 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 09:01:25.766171 6204 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 09:01:25.766321 6189 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 09:01:25.766393 6189 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781859658134510-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781859658134510-392-0/minicluster-data/ts-1/wal
uuid: "a7d0bcde5e2c4bbdbf16a4bafab7ed6a"
format_stamp: "Formatted at 2026-06-19 09:01:25 on dist-test-slave-mtdr"
server_key: "43c1947f89aff0ec13876bc46aee6310"
server_key_iv: "6f3575f03109a98867aa8923523136c6"
server_key_version: "encryptionkey@0"
I20260619 09:01:25.766462 6189 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781859658134510-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781859658134510-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781859658134510-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 09:01:25.780282 6189 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 09:01:25.782123 6189 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 09:01:25.782228 6189 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 09:01:25.782407 6189 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 09:01:25.782648 6189 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 09:01:25.782677 6189 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 09:01:25.782698 6189 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 09:01:25.782709 6189 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 09:01:25.788160 6189 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37599
I20260619 09:01:25.788213 6317 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37599 every 8 connection(s)
I20260619 09:01:25.788496 6189 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781859658134510-392-0/minicluster-data/ts-1/data/info.pb
Jun 19 09:01:25 dist-test-slave-mtdr krb5kdc[5948](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859685, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 09:01:25.797009 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ez5NB/build/release/bin/kudu as pid 6189
I20260619 09:01:25.797037 6032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:25.789172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58389 (local address 127.0.98.62:40405)
0619 09:01:25.789253 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:25.789256 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:25.789443 (+ 187us) server_negotiation.cc:407] Connection header received
0619 09:01:25.790356 (+ 913us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:25.790357 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:25.790369 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:25.790427 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:01:25.791061 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:25.791580 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:25.792535 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:25.792722 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:25.792776 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:25.794656 (+ 1880us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:01:25.794658 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:01:25.794658 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:01:25.794664 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:01:25.795876 (+ 1212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:01:25.796192 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:01:25.796192 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:01:25.796193 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:01:25.796236 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:01:25.796455 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:01:25.796457 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:01:25.796457 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:01:25.796587 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:01:25.796659 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0619 09:01:25.796901 (+ 242us) server_negotiation.cc:299] Negotiation successful
0619 09:01:25.796970 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 09:01:25.797113 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781859658134510-392-0/minicluster-data/ts-1/wal/instance
I20260619 09:01:25.797242 6320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:25.789153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40405 (local address 127.0.98.2:58389)
0619 09:01:25.789324 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0619 09:01:25.789360 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0619 09:01:25.790241 (+ 881us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:01:25.790483 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:01:25.790486 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:01:25.790733 (+ 247us) client_negotiation.cc:189] Negotiated authn=SASL
0619 09:01:25.790967 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:25.790972 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:25.791644 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:25.791644 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:25.792430 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:25.792432 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:25.792783 (+ 351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:25.792783 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:25.792872 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:25.793309 (+ 437us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 09:01:25.793317 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 09:01:25.794578 (+ 1261us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 09:01:25.795937 (+ 1359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:01:25.795941 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:01:25.795942 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:01:25.796141 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:01:25.796276 (+ 135us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:01:25.796276 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:01:25.796276 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:01:25.796373 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:01:25.796649 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 09:01:25.796652 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 09:01:25.796760 (+ 108us) client_negotiation.cc:769] Sending connection context
0619 09:01:25.796848 (+ 88us) client_negotiation.cc:240] Negotiation successful
0619 09:01:25.797006 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":93,"threads_started":1}
I20260619 09:01:25.797307 392 external_mini_cluster.cc:1428] Setting key 69ebbe55a385dac639ad41ee40c4493a
I20260619 09:01:25.797688 6318 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40405
I20260619 09:01:25.797796 6318 heartbeater.cc:464] Registering TS with master...
I20260619 09:01:25.797977 6318 heartbeater.cc:511] Master 127.0.98.62:40405 requested a full tablet report, sending...
I20260619 09:01:25.798352 5992 ts_manager.cc:194] Registered new tserver with Master: a7d0bcde5e2c4bbdbf16a4bafab7ed6a (127.0.98.2:37599)
I20260619 09:01:25.798710 5992 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:58389
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781859658134510-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781859658134510-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 09:01:25.821091 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ez5NB/build/release/bin/kudu
/tmp/dist-test-task4ez5NB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781859658134510-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781859658134510-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781859658134510-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781859658134510-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40405
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781859658134510-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39057
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781859658134510-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781859658134510-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781859658134510-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 09:01:25.898193 6323 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 09:01:25.898393 6323 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 09:01:25.898424 6323 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 09:01:25.899593 6323 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 09:01:25.899633 6323 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 09:01:25.899668 6323 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 09:01:25.901108 6323 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39057
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781859658134510-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781859658134510-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781859658134510-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781859658134510-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40405
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6323
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781859658134510-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 09:01:25.901320 6323 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 09:01:25.901520 6323 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 09:01:25.906118 6329 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:01:25.906109 6331 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:01:25.906158 6328 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:01:25.906193 6323 server_base.cc:1034] running on GCE node
I20260619 09:01:25.906582 6323 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 09:01:25.906791 6323 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 09:01:25.907970 6323 hybrid_clock.cc:648] HybridClock initialized: now 1781859685907957 us; error 27 us; skew 500 ppm
Jun 19 09:01:25 dist-test-slave-mtdr krb5kdc[5948](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859685, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 09:01:25.910318 6323 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 09:01:25.910737 6323 webserver.cc:460] Webserver started at http://127.0.98.3:43547/ using document root <none> and password file <none>
I20260619 09:01:25.910950 6323 fs_manager.cc:359] Metadata directory not provided
I20260619 09:01:25.911001 6323 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 09:01:25.911131 6323 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 09:01:25.912165 6323 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781859658134510-392-0/minicluster-data/ts-2/data/instance:
uuid: "fa92645569884d93946a0f4258a5f849"
format_stamp: "Formatted at 2026-06-19 09:01:25 on dist-test-slave-mtdr"
server_key: "4a38adc29d348a83a00faf9939f611eb"
server_key_iv: "429aa5122424bc353c5f53b5f8ba4a5e"
server_key_version: "encryptionkey@0"
I20260619 09:01:25.912518 6323 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781859658134510-392-0/minicluster-data/ts-2/wal/instance:
uuid: "fa92645569884d93946a0f4258a5f849"
format_stamp: "Formatted at 2026-06-19 09:01:25 on dist-test-slave-mtdr"
server_key: "4a38adc29d348a83a00faf9939f611eb"
server_key_iv: "429aa5122424bc353c5f53b5f8ba4a5e"
server_key_version: "encryptionkey@0"
I20260619 09:01:25.913770 6323 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 09:01:25.914402 6338 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 09:01:25.914539 6323 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 09:01:25.914615 6323 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781859658134510-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781859658134510-392-0/minicluster-data/ts-2/wal
uuid: "fa92645569884d93946a0f4258a5f849"
format_stamp: "Formatted at 2026-06-19 09:01:25 on dist-test-slave-mtdr"
server_key: "4a38adc29d348a83a00faf9939f611eb"
server_key_iv: "429aa5122424bc353c5f53b5f8ba4a5e"
server_key_version: "encryptionkey@0"
I20260619 09:01:25.914672 6323 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781859658134510-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781859658134510-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781859658134510-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 09:01:25.926651 6323 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 09:01:25.928555 6323 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 09:01:25.928678 6323 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 09:01:25.928862 6323 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 09:01:25.929136 6323 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 09:01:25.929169 6323 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 09:01:25.929195 6323 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 09:01:25.929219 6323 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 09:01:25.934469 6323 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36503
I20260619 09:01:25.934526 6451 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36503 every 8 connection(s)
I20260619 09:01:25.934801 6323 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781859658134510-392-0/minicluster-data/ts-2/data/info.pb
I20260619 09:01:25.935237 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ez5NB/build/release/bin/kudu as pid 6323
I20260619 09:01:25.935314 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781859658134510-392-0/minicluster-data/ts-2/wal/instance
I20260619 09:01:25.935489 392 external_mini_cluster.cc:1428] Setting key 601287e8b71ea0a98a2585b313dc3bc1
Jun 19 09:01:25 dist-test-slave-mtdr krb5kdc[5948](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859685, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 09:01:25.943708 6032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:25.935586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51767 (local address 127.0.98.62:40405)
0619 09:01:25.935640 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:25.935642 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:25.935848 (+ 206us) server_negotiation.cc:407] Connection header received
0619 09:01:25.936871 (+ 1023us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:25.936872 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:25.936885 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:25.936934 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:01:25.937551 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:25.938056 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:25.939060 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:25.939262 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:25.939324 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:25.941330 (+ 2006us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:01:25.941332 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:01:25.941332 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:01:25.941338 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:01:25.942600 (+ 1262us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:01:25.943041 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:01:25.943042 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:01:25.943042 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:01:25.943097 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:01:25.943267 (+ 170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:01:25.943267 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:01:25.943267 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:01:25.943358 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:01:25.943399 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0619 09:01:25.943592 (+ 193us) server_negotiation.cc:299] Negotiation successful
0619 09:01:25.943645 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260619 09:01:25.943859 6454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:25.935514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40405 (local address 127.0.98.3:51767)
0619 09:01:25.935705 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0619 09:01:25.935746 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 09:01:25.936768 (+ 1022us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:01:25.937018 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:01:25.937023 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:01:25.937275 (+ 252us) client_negotiation.cc:189] Negotiated authn=SASL
0619 09:01:25.937459 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:25.937463 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:25.938141 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:25.938142 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:25.938974 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:25.938977 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:25.939334 (+ 357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:25.939335 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:25.939458 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:25.939896 (+ 438us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 09:01:25.939905 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0619 09:01:25.941235 (+ 1330us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 09:01:25.942670 (+ 1435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:01:25.942675 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:01:25.942677 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:01:25.942970 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:01:25.943143 (+ 173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:01:25.943143 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:01:25.943144 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:01:25.943231 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:01:25.943396 (+ 165us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 09:01:25.943399 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 09:01:25.943500 (+ 101us) client_negotiation.cc:769] Sending connection context
0619 09:01:25.943579 (+ 79us) client_negotiation.cc:240] Negotiation successful
0619 09:01:25.943684 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":80,"threads_started":1}
I20260619 09:01:25.944262 6452 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40405
I20260619 09:01:25.944346 6452 heartbeater.cc:464] Registering TS with master...
I20260619 09:01:25.944506 6452 heartbeater.cc:511] Master 127.0.98.62:40405 requested a full tablet report, sending...
I20260619 09:01:25.944800 5992 ts_manager.cc:194] Registered new tserver with Master: fa92645569884d93946a0f4258a5f849 (127.0.98.3:36503)
I20260619 09:01:25.945134 5992 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:51767
I20260619 09:01:25.947057 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781859658134510-392-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781859658134510-392-0/krb5kdc/test-admin.keytab.
Jun 19 09:01:25 dist-test-slave-mtdr krb5kdc[5948](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859685, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 09:01:25.960161 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260619 09:01:25.961499 6459 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 09:01:25.960878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60714 (local address 127.0.98.1:46667)
0619 09:01:25.961025 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:25.961029 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:25.961038 (+ 9us) server_negotiation.cc:407] Connection header received
0619 09:01:25.961061 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:25.961063 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:25.961078 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:25.961143 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:01:25.961427 (+ 284us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60714: BlockingRecv error: recv got EOF from 127.0.0.1:60714 (error 108)
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":74,"threads_started":1}
W20260619 09:01:25.961689 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781859658134510-392-0/krb5kdc/krb5cc' of type FILE
Jun 19 09:01:25 dist-test-slave-mtdr krb5kdc[5948](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859685, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 09:01:25.962486 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 19 09:01:25 dist-test-slave-mtdr krb5kdc[5948](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859685, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 09:01:25.963330 392 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 19 09:01:25 dist-test-slave-mtdr krb5kdc[5948](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859685, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260619 09:01:25.970235 6459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:25.963805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60716 (local address 127.0.98.1:46667)
0619 09:01:25.963853 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:25.963855 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:25.963863 (+ 8us) server_negotiation.cc:407] Connection header received
0619 09:01:25.963913 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:25.963914 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:25.963925 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:25.963976 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:01:25.964325 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:25.965027 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:25.965686 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:25.965985 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:25.966042 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:25.967855 (+ 1813us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:01:25.967863 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:01:25.967863 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:01:25.967870 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:01:25.969174 (+ 1304us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:01:25.969469 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:01:25.969472 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:01:25.969473 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:01:25.969549 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:01:25.969713 (+ 164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:01:25.969715 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:01:25.969723 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:01:25.969848 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:01:25.969962 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0619 09:01:25.970076 (+ 114us) server_negotiation.cc:299] Negotiation successful
0619 09:01:25.970141 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
W20260619 09:01:25.971873 6459 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 09:01:25.971335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60728 (local address 127.0.98.1:46667)
0619 09:01:25.971368 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:25.971369 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:25.971448 (+ 79us) server_negotiation.cc:407] Connection header received
0619 09:01:25.971502 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:25.971503 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:25.971512 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:25.971586 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:01:25.971830 (+ 244us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60728: BlockingRecv error: recv got EOF from 127.0.0.1:60728 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260619 09:01:25.971926 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781859658134510-392-0/krb5kdc/krb5cc' of type FILE
Jun 19 09:01:25 dist-test-slave-mtdr krb5kdc[5948](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859685, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 09:01:25.972724 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 19 09:01:25 dist-test-slave-mtdr krb5kdc[5948](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859685, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 09:01:25.973430 392 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 19 09:01:25 dist-test-slave-mtdr krb5kdc[5948](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859685, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260619 09:01:25.979400 6459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:25.973872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60736 (local address 127.0.98.1:46667)
0619 09:01:25.973938 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:25.973940 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:25.973949 (+ 9us) server_negotiation.cc:407] Connection header received
0619 09:01:25.973987 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:25.973987 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:25.973996 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:25.974047 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:01:25.974332 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:25.974896 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:25.975540 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:25.975731 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:25.975789 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:25.977561 (+ 1772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:01:25.977562 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:01:25.977563 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:01:25.977568 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:01:25.978586 (+ 1018us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:01:25.978895 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:01:25.978896 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:01:25.978896 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:01:25.978943 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:01:25.979075 (+ 132us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:01:25.979075 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:01:25.979076 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:01:25.979148 (+ 72us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:01:25.979205 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0619 09:01:25.979274 (+ 69us) server_negotiation.cc:299] Negotiation successful
0619 09:01:25.979326 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
W20260619 09:01:25.980618 6459 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 09:01:25.980220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60742 (local address 127.0.98.1:46667)
0619 09:01:25.980270 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:25.980272 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:25.980281 (+ 9us) server_negotiation.cc:407] Connection header received
0619 09:01:25.980320 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:25.980320 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:25.980327 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:25.980365 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:01:25.980579 (+ 214us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60742: BlockingRecv error: recv got EOF from 127.0.0.1:60742 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
W20260619 09:01:25.980670 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781859658134510-392-0/krb5kdc/krb5cc' of type FILE
Jun 19 09:01:25 dist-test-slave-mtdr krb5kdc[5948](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859685, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 09:01:25.981488 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 19 09:01:25 dist-test-slave-mtdr krb5kdc[5948](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859685, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 09:01:25.982177 392 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 19 09:01:25 dist-test-slave-mtdr krb5kdc[5948](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859685, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260619 09:01:25.987835 6459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:25.982585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60748 (local address 127.0.98.1:46667)
0619 09:01:25.982621 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:25.982622 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:25.982668 (+ 46us) server_negotiation.cc:407] Connection header received
0619 09:01:25.982739 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:25.982739 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:25.982746 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:25.982782 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:01:25.983126 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:25.983554 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:25.984165 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:25.984335 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:25.984386 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:25.986087 (+ 1701us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:01:25.986088 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:01:25.986088 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:01:25.986093 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:01:25.987118 (+ 1025us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:01:25.987373 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:01:25.987374 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:01:25.987374 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:01:25.987413 (+ 39us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:01:25.987532 (+ 119us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:01:25.987533 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:01:25.987533 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:01:25.987601 (+ 68us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:01:25.987669 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0619 09:01:25.987736 (+ 67us) server_negotiation.cc:299] Negotiation successful
0619 09:01:25.987783 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260619 09:01:25.988307 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ez5NB/build/release/bin/kudu with pid 6055
I20260619 09:01:25.992427 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ez5NB/build/release/bin/kudu with pid 6189
I20260619 09:01:25.995951 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ez5NB/build/release/bin/kudu with pid 6323
I20260619 09:01:25.999508 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ez5NB/build/release/bin/kudu with pid 5961
2026-06-19T09:01:26Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (1752 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859658134510-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 09:01:26 dist-test-slave-mtdr krb5kdc[6472](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 09:01:26 dist-test-slave-mtdr krb5kdc[6472](info): set up 2 sockets
Jun 19 09:01:26 dist-test-slave-mtdr krb5kdc[6472](info): commencing operation
krb5kdc: starting...
W20260619 09:01:26.950958 392 mini_kdc.cc:121] Time spent starting KDC: real 0.933s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859658134510-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859658134510-392-0/krb5kdc/test-user.keytab.
Jun 19 09:01:27 dist-test-slave-mtdr krb5kdc[6472](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859687, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T09:01:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T09:01:27Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859658134510-392-0/krb5kdc/oryx_127.0.98.62.keytab.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859658134510-392-0/krb5kdc/oryx_127.0.98.62.keytab.
I20260619 09:01:27.058662 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ez5NB/build/release/bin/kudu
/tmp/dist-test-task4ez5NB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859658134510-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859658134510-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859658134510-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859658134510-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44403
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859658134510-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42929
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44403
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859658134510-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859658134510-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859658134510-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 09:01:27.135702 6485 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 09:01:27.135895 6485 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 09:01:27.135915 6485 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 09:01:27.136986 6485 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 09:01:27.137029 6485 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 09:01:27.137041 6485 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 09:01:27.137053 6485 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 09:01:27.137063 6485 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 09:01:27.138484 6485 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42929
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859658134510-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859658134510-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44403
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859658134510-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44403
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859658134510-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6485
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859658134510-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 09:01:27.138667 6485 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 09:01:27.138880 6485 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 09:01:27.143168 6491 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:01:27.143210 6490 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:01:27.143384 6485 server_base.cc:1034] running on GCE node
W20260619 09:01:27.143179 6493 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:01:27.143651 6485 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 09:01:27.143930 6485 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 09:01:27.145064 6485 hybrid_clock.cc:648] HybridClock initialized: now 1781859687145056 us; error 35 us; skew 500 ppm
Jun 19 09:01:27 dist-test-slave-mtdr krb5kdc[6472](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859687, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 09:01:27.147382 6485 init.cc:377] Logged in from keytab as oryx/127.0.98.62@KRBTEST.COM (short username oryx)
I20260619 09:01:27.147744 6485 webserver.cc:460] Webserver started at http://127.0.98.62:33335/ using document root <none> and password file <none>
I20260619 09:01:27.147934 6485 fs_manager.cc:359] Metadata directory not provided
I20260619 09:01:27.147974 6485 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 09:01:27.148059 6485 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 09:01:27.149019 6485 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859658134510-392-0/minicluster-data/master-0/data/instance:
uuid: "aaa8435622414a159f995f651c058e08"
format_stamp: "Formatted at 2026-06-19 09:01:27 on dist-test-slave-mtdr"
server_key: "c0e8c8963d0ff7be3a2285b272a46bb1"
server_key_iv: "77a5d337fc07b0f7c504bfac76d10736"
server_key_version: "encryptionkey@0"
I20260619 09:01:27.149298 6485 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859658134510-392-0/minicluster-data/master-0/wal/instance:
uuid: "aaa8435622414a159f995f651c058e08"
format_stamp: "Formatted at 2026-06-19 09:01:27 on dist-test-slave-mtdr"
server_key: "c0e8c8963d0ff7be3a2285b272a46bb1"
server_key_iv: "77a5d337fc07b0f7c504bfac76d10736"
server_key_version: "encryptionkey@0"
I20260619 09:01:27.150437 6485 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 09:01:27.151091 6500 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 09:01:27.151240 6485 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 09:01:27.151319 6485 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859658134510-392-0/minicluster-data/master-0/data,/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859658134510-392-0/minicluster-data/master-0/wal
uuid: "aaa8435622414a159f995f651c058e08"
format_stamp: "Formatted at 2026-06-19 09:01:27 on dist-test-slave-mtdr"
server_key: "c0e8c8963d0ff7be3a2285b272a46bb1"
server_key_iv: "77a5d337fc07b0f7c504bfac76d10736"
server_key_version: "encryptionkey@0"
I20260619 09:01:27.151376 6485 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859658134510-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859658134510-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859658134510-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 09:01:27.176904 6485 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 09:01:27.178778 6485 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 09:01:27.178931 6485 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 09:01:27.182459 6485 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44403
I20260619 09:01:27.182514 6552 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44403 every 8 connection(s)
I20260619 09:01:27.182811 6485 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859658134510-392-0/minicluster-data/master-0/data/info.pb
I20260619 09:01:27.183768 6553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 09:01:27.185930 6553 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aaa8435622414a159f995f651c058e08: Bootstrap starting.
I20260619 09:01:27.186451 6553 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aaa8435622414a159f995f651c058e08: Neither blocks nor log segments found. Creating new log.
I20260619 09:01:27.186702 6553 log.cc:824] T 00000000000000000000000000000000 P aaa8435622414a159f995f651c058e08: Log is configured to *not* fsync() on all Append() calls
I20260619 09:01:27.187335 6553 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aaa8435622414a159f995f651c058e08: No bootstrap required, opened a new log
I20260619 09:01:27.188954 6553 raft_consensus.cc:348] T 00000000000000000000000000000000 P aaa8435622414a159f995f651c058e08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aaa8435622414a159f995f651c058e08" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44403 } }
I20260619 09:01:27.189086 6553 raft_consensus.cc:374] T 00000000000000000000000000000000 P aaa8435622414a159f995f651c058e08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 09:01:27.189107 6553 raft_consensus.cc:729] T 00000000000000000000000000000000 P aaa8435622414a159f995f651c058e08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aaa8435622414a159f995f651c058e08, State: Initialized, Role: FOLLOWER
I20260619 09:01:27.189235 6553 consensus_queue.cc:260] T 00000000000000000000000000000000 P aaa8435622414a159f995f651c058e08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aaa8435622414a159f995f651c058e08" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44403 } }
I20260619 09:01:27.189298 6553 raft_consensus.cc:388] T 00000000000000000000000000000000 P aaa8435622414a159f995f651c058e08 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 09:01:27.189319 6553 raft_consensus.cc:482] T 00000000000000000000000000000000 P aaa8435622414a159f995f651c058e08 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 09:01:27.189347 6553 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aaa8435622414a159f995f651c058e08 [term 0 FOLLOWER]: Advancing to term 1
I20260619 09:01:27.190088 6553 raft_consensus.cc:504] T 00000000000000000000000000000000 P aaa8435622414a159f995f651c058e08 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aaa8435622414a159f995f651c058e08" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44403 } }
I20260619 09:01:27.190176 6553 leader_election.cc:302] T 00000000000000000000000000000000 P aaa8435622414a159f995f651c058e08 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aaa8435622414a159f995f651c058e08; no voters:
I20260619 09:01:27.190317 6553 leader_election.cc:288] T 00000000000000000000000000000000 P aaa8435622414a159f995f651c058e08 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 09:01:27.190349 6556 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aaa8435622414a159f995f651c058e08 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 09:01:27.190464 6556 raft_consensus.cc:686] T 00000000000000000000000000000000 P aaa8435622414a159f995f651c058e08 [term 1 LEADER]: Becoming Leader. State: Replica: aaa8435622414a159f995f651c058e08, State: Running, Role: LEADER
I20260619 09:01:27.190557 6553 sys_catalog.cc:564] T 00000000000000000000000000000000 P aaa8435622414a159f995f651c058e08 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 09:01:27.190549 6556 consensus_queue.cc:237] T 00000000000000000000000000000000 P aaa8435622414a159f995f651c058e08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aaa8435622414a159f995f651c058e08" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44403 } }
I20260619 09:01:27.191020 6558 sys_catalog.cc:455] T 00000000000000000000000000000000 P aaa8435622414a159f995f651c058e08 [sys.catalog]: SysCatalogTable state changed. Reason: New leader aaa8435622414a159f995f651c058e08. Latest consensus state: current_term: 1 leader_uuid: "aaa8435622414a159f995f651c058e08" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aaa8435622414a159f995f651c058e08" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44403 } } }
I20260619 09:01:27.191007 6557 sys_catalog.cc:455] T 00000000000000000000000000000000 P aaa8435622414a159f995f651c058e08 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "aaa8435622414a159f995f651c058e08" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aaa8435622414a159f995f651c058e08" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44403 } } }
I20260619 09:01:27.191104 6558 sys_catalog.cc:458] T 00000000000000000000000000000000 P aaa8435622414a159f995f651c058e08 [sys.catalog]: This master's current role is: LEADER
I20260619 09:01:27.191109 6557 sys_catalog.cc:458] T 00000000000000000000000000000000 P aaa8435622414a159f995f651c058e08 [sys.catalog]: This master's current role is: LEADER
I20260619 09:01:27.191300 6564 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 09:01:27.191776 6564 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 09:01:27.192703 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ez5NB/build/release/bin/kudu as pid 6485
I20260619 09:01:27.192819 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859658134510-392-0/minicluster-data/master-0/wal/instance
I20260619 09:01:27.193011 392 external_mini_cluster.cc:1428] Setting key eac2e2bc1725dd941008af98588e419b
I20260619 09:01:27.193334 6564 catalog_manager.cc:1347] Generated new cluster ID: aac0526a83f348348927f89c7ec729d5
I20260619 09:01:27.193382 6564 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 19 09:01:27 dist-test-slave-mtdr krb5kdc[6472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859687, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260619 09:01:27.202699 6564 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 09:01:27.202968 6576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:27.193427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43014 (local address 127.0.98.62:44403)
0619 09:01:27.194927 (+ 1500us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:27.194935 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:27.194959 (+ 24us) server_negotiation.cc:407] Connection header received
0619 09:01:27.195053 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:27.195057 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:27.195098 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:27.195210 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:01:27.195781 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:27.196672 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:27.197379 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:27.197785 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:27.197878 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:27.199975 (+ 2097us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:01:27.199979 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:01:27.199979 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:01:27.199990 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:01:27.201443 (+ 1453us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:01:27.201740 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:01:27.201743 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:01:27.201743 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:01:27.201838 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:01:27.202006 (+ 168us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:01:27.202007 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:01:27.202007 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:01:27.202230 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:01:27.202382 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0619 09:01:27.202476 (+ 94us) server_negotiation.cc:299] Negotiation successful
0619 09:01:27.202612 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":1459,"thread_start_us":69,"threads_started":1}
I20260619 09:01:27.203395 6564 catalog_manager.cc:1504] Loading token signing keys...
I20260619 09:01:27.212179 6564 catalog_manager.cc:5868] T 00000000000000000000000000000000 P aaa8435622414a159f995f651c058e08: Generated new TSK 0
I20260619 09:01:27.212356 6564 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859658134510-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859658134510-392-0/krb5kdc/oryx_127.0.98.1.keytab.
I20260619 09:01:27.235363 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ez5NB/build/release/bin/kudu
/tmp/dist-test-task4ez5NB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859658134510-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859658134510-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859658134510-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859658134510-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44403
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859658134510-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42929
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859658134510-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859658134510-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859658134510-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 09:01:27.312955 6579 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 09:01:27.313154 6579 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 09:01:27.313175 6579 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 09:01:27.314281 6579 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 09:01:27.314320 6579 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 09:01:27.314344 6579 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 09:01:27.315809 6579 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42929
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859658134510-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859658134510-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859658134510-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859658134510-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44403
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6579
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859658134510-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 09:01:27.316025 6579 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 09:01:27.316227 6579 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 09:01:27.320895 6584 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:01:27.320897 6587 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:01:27.321089 6585 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:01:27.321079 6579 server_base.cc:1034] running on GCE node
I20260619 09:01:27.321285 6579 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 09:01:27.321480 6579 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 09:01:27.322640 6579 hybrid_clock.cc:648] HybridClock initialized: now 1781859687322627 us; error 26 us; skew 500 ppm
Jun 19 09:01:27 dist-test-slave-mtdr krb5kdc[6472](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859687, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 09:01:27.325047 6579 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260619 09:01:27.325465 6579 webserver.cc:460] Webserver started at http://127.0.98.1:37141/ using document root <none> and password file <none>
I20260619 09:01:27.325659 6579 fs_manager.cc:359] Metadata directory not provided
I20260619 09:01:27.325700 6579 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 09:01:27.325812 6579 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 09:01:27.326792 6579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859658134510-392-0/minicluster-data/ts-0/data/instance:
uuid: "551feb9174794395a270058ce5ed45c5"
format_stamp: "Formatted at 2026-06-19 09:01:27 on dist-test-slave-mtdr"
server_key: "900dbdbbd43114037e85d9c7fa4444d0"
server_key_iv: "8baf4b3c5db85ad229d3e9464408e366"
server_key_version: "encryptionkey@0"
I20260619 09:01:27.327145 6579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859658134510-392-0/minicluster-data/ts-0/wal/instance:
uuid: "551feb9174794395a270058ce5ed45c5"
format_stamp: "Formatted at 2026-06-19 09:01:27 on dist-test-slave-mtdr"
server_key: "900dbdbbd43114037e85d9c7fa4444d0"
server_key_iv: "8baf4b3c5db85ad229d3e9464408e366"
server_key_version: "encryptionkey@0"
I20260619 09:01:27.328402 6579 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 09:01:27.329095 6594 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 09:01:27.329257 6579 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 09:01:27.329377 6579 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859658134510-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859658134510-392-0/minicluster-data/ts-0/wal
uuid: "551feb9174794395a270058ce5ed45c5"
format_stamp: "Formatted at 2026-06-19 09:01:27 on dist-test-slave-mtdr"
server_key: "900dbdbbd43114037e85d9c7fa4444d0"
server_key_iv: "8baf4b3c5db85ad229d3e9464408e366"
server_key_version: "encryptionkey@0"
I20260619 09:01:27.329443 6579 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859658134510-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859658134510-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859658134510-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 09:01:27.340099 6579 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 09:01:27.341742 6579 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 09:01:27.341864 6579 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 09:01:27.342051 6579 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 09:01:27.342319 6579 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 09:01:27.342360 6579 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 09:01:27.342387 6579 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 09:01:27.342406 6579 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 09:01:27.347817 6579 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40909
I20260619 09:01:27.347882 6707 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40909 every 8 connection(s)
I20260619 09:01:27.348153 6579 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859658134510-392-0/minicluster-data/ts-0/data/info.pb
I20260619 09:01:27.349660 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ez5NB/build/release/bin/kudu as pid 6579
I20260619 09:01:27.349745 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859658134510-392-0/minicluster-data/ts-0/wal/instance
I20260619 09:01:27.349921 392 external_mini_cluster.cc:1428] Setting key ba279791fe1b3e2954aff3edd06e6efa
Jun 19 09:01:27 dist-test-slave-mtdr krb5kdc[6472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859687, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260619 09:01:27.357206 6576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:27.348810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45975 (local address 127.0.98.62:44403)
0619 09:01:27.348865 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:27.348867 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:27.349038 (+ 171us) server_negotiation.cc:407] Connection header received
0619 09:01:27.349554 (+ 516us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:27.349555 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:27.349571 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:27.349622 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:01:27.350640 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:27.351190 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:27.352533 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:27.352735 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:27.352794 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:27.354731 (+ 1937us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:01:27.354733 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:01:27.354733 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:01:27.354741 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:01:27.356006 (+ 1265us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:01:27.356354 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:01:27.356356 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:01:27.356356 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:01:27.356413 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:01:27.356627 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:01:27.356628 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:01:27.356628 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:01:27.356737 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:01:27.356790 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0619 09:01:27.357036 (+ 246us) server_negotiation.cc:299] Negotiation successful
0619 09:01:27.357118 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260619 09:01:27.357374 6710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:27.348775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44403 (local address 127.0.98.1:45975)
0619 09:01:27.348930 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0619 09:01:27.348963 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0619 09:01:27.349441 (+ 478us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:01:27.349693 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:01:27.349696 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:01:27.350339 (+ 643us) client_negotiation.cc:189] Negotiated authn=SASL
0619 09:01:27.350561 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:27.350568 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:27.351274 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:27.351275 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:27.352446 (+ 1171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:27.352449 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:27.352802 (+ 353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:27.352803 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:27.352909 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:27.353371 (+ 462us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 09:01:27.353379 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 09:01:27.354646 (+ 1267us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 09:01:27.356080 (+ 1434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:01:27.356087 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:01:27.356089 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:01:27.356290 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:01:27.356464 (+ 174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:01:27.356464 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:01:27.356465 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:01:27.356572 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:01:27.356789 (+ 217us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 09:01:27.356795 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 09:01:27.356924 (+ 129us) client_negotiation.cc:769] Sending connection context
0619 09:01:27.357009 (+ 85us) client_negotiation.cc:240] Negotiation successful
0619 09:01:27.357167 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":78,"threads_started":1}
I20260619 09:01:27.357806 6708 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44403
I20260619 09:01:27.357889 6708 heartbeater.cc:464] Registering TS with master...
I20260619 09:01:27.358063 6708 heartbeater.cc:511] Master 127.0.98.62:44403 requested a full tablet report, sending...
I20260619 09:01:27.358562 6517 ts_manager.cc:194] Registered new tserver with Master: 551feb9174794395a270058ce5ed45c5 (127.0.98.1:40909)
I20260619 09:01:27.359251 6517 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:45975
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859658134510-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859658134510-392-0/krb5kdc/oryx_127.0.98.2.keytab.
I20260619 09:01:27.371644 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ez5NB/build/release/bin/kudu
/tmp/dist-test-task4ez5NB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859658134510-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859658134510-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859658134510-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859658134510-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44403
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859658134510-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42929
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859658134510-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859658134510-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859658134510-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 09:01:27.448375 6713 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 09:01:27.448570 6713 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 09:01:27.448591 6713 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 09:01:27.449681 6713 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 09:01:27.449717 6713 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 09:01:27.449743 6713 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 09:01:27.451153 6713 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42929
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859658134510-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859658134510-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859658134510-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859658134510-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44403
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6713
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859658134510-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 09:01:27.451357 6713 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 09:01:27.451555 6713 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 09:01:27.456082 6721 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:01:27.456075 6718 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:01:27.456149 6719 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:01:27.456310 6713 server_base.cc:1034] running on GCE node
I20260619 09:01:27.456475 6713 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 09:01:27.456691 6713 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 09:01:27.457849 6713 hybrid_clock.cc:648] HybridClock initialized: now 1781859687457835 us; error 28 us; skew 500 ppm
Jun 19 09:01:27 dist-test-slave-mtdr krb5kdc[6472](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859687, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 09:01:27.460073 6713 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260619 09:01:27.460474 6713 webserver.cc:460] Webserver started at http://127.0.98.2:34281/ using document root <none> and password file <none>
I20260619 09:01:27.460680 6713 fs_manager.cc:359] Metadata directory not provided
I20260619 09:01:27.460726 6713 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 09:01:27.460846 6713 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 09:01:27.461826 6713 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859658134510-392-0/minicluster-data/ts-1/data/instance:
uuid: "00cca0af6ec34749baea4d021c20755e"
format_stamp: "Formatted at 2026-06-19 09:01:27 on dist-test-slave-mtdr"
server_key: "247defef2043a8dcc0ac213830a070ef"
server_key_iv: "7e262f62ec1186890ade9b148212dbe3"
server_key_version: "encryptionkey@0"
I20260619 09:01:27.462131 6713 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859658134510-392-0/minicluster-data/ts-1/wal/instance:
uuid: "00cca0af6ec34749baea4d021c20755e"
format_stamp: "Formatted at 2026-06-19 09:01:27 on dist-test-slave-mtdr"
server_key: "247defef2043a8dcc0ac213830a070ef"
server_key_iv: "7e262f62ec1186890ade9b148212dbe3"
server_key_version: "encryptionkey@0"
I20260619 09:01:27.463394 6713 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 09:01:27.464044 6728 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 09:01:27.464187 6713 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 09:01:27.464263 6713 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859658134510-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859658134510-392-0/minicluster-data/ts-1/wal
uuid: "00cca0af6ec34749baea4d021c20755e"
format_stamp: "Formatted at 2026-06-19 09:01:27 on dist-test-slave-mtdr"
server_key: "247defef2043a8dcc0ac213830a070ef"
server_key_iv: "7e262f62ec1186890ade9b148212dbe3"
server_key_version: "encryptionkey@0"
I20260619 09:01:27.464316 6713 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859658134510-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859658134510-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859658134510-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 09:01:27.491952 6713 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 09:01:27.493727 6713 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 09:01:27.493854 6713 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 09:01:27.494058 6713 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 09:01:27.494333 6713 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 09:01:27.494364 6713 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 09:01:27.494392 6713 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 09:01:27.494415 6713 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 09:01:27.499655 6713 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37741
I20260619 09:01:27.499706 6841 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37741 every 8 connection(s)
I20260619 09:01:27.499980 6713 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859658134510-392-0/minicluster-data/ts-1/data/info.pb
Jun 19 09:01:27 dist-test-slave-mtdr krb5kdc[6472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859687, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260619 09:01:27.505613 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ez5NB/build/release/bin/kudu as pid 6713
I20260619 09:01:27.505686 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859658134510-392-0/minicluster-data/ts-1/wal/instance
I20260619 09:01:27.505857 392 external_mini_cluster.cc:1428] Setting key 0e57c5c50a6982f6ea860b121a8a5ac5
I20260619 09:01:27.508397 6576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:27.500601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34123 (local address 127.0.98.62:44403)
0619 09:01:27.500667 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:27.500669 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:27.500848 (+ 179us) server_negotiation.cc:407] Connection header received
0619 09:01:27.501310 (+ 462us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:27.501311 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:27.501320 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:27.501371 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:01:27.502229 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:27.502699 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:27.503670 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:27.503845 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:27.503892 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:27.505766 (+ 1874us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:01:27.505768 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:01:27.505769 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:01:27.505776 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:01:27.507199 (+ 1423us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:01:27.507535 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:01:27.507536 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:01:27.507536 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:01:27.507591 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:01:27.507805 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:01:27.507806 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:01:27.507807 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:01:27.507912 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:01:27.507971 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0619 09:01:27.508221 (+ 250us) server_negotiation.cc:299] Negotiation successful
0619 09:01:27.508301 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260619 09:01:27.508591 6844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:27.500575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44403 (local address 127.0.98.2:34123)
0619 09:01:27.500730 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0619 09:01:27.500770 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 09:01:27.501233 (+ 463us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:01:27.501434 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:01:27.501437 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:01:27.501971 (+ 534us) client_negotiation.cc:189] Negotiated authn=SASL
0619 09:01:27.502162 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:27.502167 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:27.502758 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:27.502759 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:27.503584 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:27.503586 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:27.503900 (+ 314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:27.503900 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:27.503992 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:27.504428 (+ 436us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 09:01:27.504436 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 09:01:27.505678 (+ 1242us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 09:01:27.507265 (+ 1587us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:01:27.507271 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:01:27.507273 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:01:27.507465 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:01:27.507649 (+ 184us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:01:27.507649 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:01:27.507649 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:01:27.507732 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:01:27.507969 (+ 237us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 09:01:27.507975 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 09:01:27.508104 (+ 129us) client_negotiation.cc:769] Sending connection context
0619 09:01:27.508191 (+ 87us) client_negotiation.cc:240] Negotiation successful
0619 09:01:27.508344 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":84,"threads_started":1}
I20260619 09:01:27.509006 6842 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44403
I20260619 09:01:27.509094 6842 heartbeater.cc:464] Registering TS with master...
I20260619 09:01:27.509254 6842 heartbeater.cc:511] Master 127.0.98.62:44403 requested a full tablet report, sending...
I20260619 09:01:27.509582 6517 ts_manager.cc:194] Registered new tserver with Master: 00cca0af6ec34749baea4d021c20755e (127.0.98.2:37741)
I20260619 09:01:27.509950 6517 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:34123
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859658134510-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859658134510-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260619 09:01:27.527521 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ez5NB/build/release/bin/kudu
/tmp/dist-test-task4ez5NB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859658134510-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859658134510-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859658134510-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859658134510-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44403
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859658134510-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42929
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859658134510-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859658134510-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859658134510-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 09:01:27.604838 6847 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 09:01:27.605033 6847 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 09:01:27.605054 6847 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 09:01:27.606117 6847 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 09:01:27.606148 6847 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 09:01:27.606173 6847 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 09:01:27.607543 6847 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42929
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859658134510-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859658134510-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859658134510-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859658134510-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44403
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6847
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859658134510-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 09:01:27.607722 6847 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 09:01:27.607897 6847 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 09:01:27.612376 6855 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:01:27.612457 6853 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:01:27.612558 6847 server_base.cc:1034] running on GCE node
W20260619 09:01:27.612568 6852 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:01:27.612810 6847 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 09:01:27.613032 6847 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 09:01:27.614166 6847 hybrid_clock.cc:648] HybridClock initialized: now 1781859687614149 us; error 31 us; skew 500 ppm
Jun 19 09:01:27 dist-test-slave-mtdr krb5kdc[6472](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859687, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 09:01:27.616707 6847 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260619 09:01:27.617138 6847 webserver.cc:460] Webserver started at http://127.0.98.3:46723/ using document root <none> and password file <none>
I20260619 09:01:27.617311 6847 fs_manager.cc:359] Metadata directory not provided
I20260619 09:01:27.617347 6847 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 09:01:27.617434 6847 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 09:01:27.618393 6847 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859658134510-392-0/minicluster-data/ts-2/data/instance:
uuid: "a5cc45b850424e7da92acf9089450d80"
format_stamp: "Formatted at 2026-06-19 09:01:27 on dist-test-slave-mtdr"
server_key: "80071be4a7876c182c9048c1acaceb93"
server_key_iv: "f3398164a79243eed432d8f88b31b9d5"
server_key_version: "encryptionkey@0"
I20260619 09:01:27.618676 6847 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859658134510-392-0/minicluster-data/ts-2/wal/instance:
uuid: "a5cc45b850424e7da92acf9089450d80"
format_stamp: "Formatted at 2026-06-19 09:01:27 on dist-test-slave-mtdr"
server_key: "80071be4a7876c182c9048c1acaceb93"
server_key_iv: "f3398164a79243eed432d8f88b31b9d5"
server_key_version: "encryptionkey@0"
I20260619 09:01:27.619818 6847 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260619 09:01:27.620415 6862 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 09:01:27.620546 6847 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 09:01:27.620646 6847 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859658134510-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859658134510-392-0/minicluster-data/ts-2/wal
uuid: "a5cc45b850424e7da92acf9089450d80"
format_stamp: "Formatted at 2026-06-19 09:01:27 on dist-test-slave-mtdr"
server_key: "80071be4a7876c182c9048c1acaceb93"
server_key_iv: "f3398164a79243eed432d8f88b31b9d5"
server_key_version: "encryptionkey@0"
I20260619 09:01:27.620707 6847 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859658134510-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859658134510-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859658134510-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 09:01:27.633497 6847 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 09:01:27.635094 6847 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 09:01:27.635210 6847 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 09:01:27.635396 6847 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 09:01:27.635670 6847 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 09:01:27.635701 6847 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 09:01:27.635728 6847 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 09:01:27.635752 6847 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 09:01:27.640889 6847 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34875
I20260619 09:01:27.640944 6975 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34875 every 8 connection(s)
I20260619 09:01:27.641237 6847 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859658134510-392-0/minicluster-data/ts-2/data/info.pb
I20260619 09:01:27.641559 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ez5NB/build/release/bin/kudu as pid 6847
I20260619 09:01:27.641638 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859658134510-392-0/minicluster-data/ts-2/wal/instance
I20260619 09:01:27.641817 392 external_mini_cluster.cc:1428] Setting key aa2d31ce8dad463206ba62eb8686c1b9
Jun 19 09:01:27 dist-test-slave-mtdr krb5kdc[6472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859687, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260619 09:01:27.649647 6576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:27.641892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55349 (local address 127.0.98.62:44403)
0619 09:01:27.641950 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:27.641953 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:27.642178 (+ 225us) server_negotiation.cc:407] Connection header received
0619 09:01:27.642720 (+ 542us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:27.642721 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:27.642731 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:27.642774 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:01:27.643641 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:27.644119 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:27.645327 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:27.645568 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:27.645612 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:27.647493 (+ 1881us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:01:27.647494 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:01:27.647494 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:01:27.647499 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:01:27.648686 (+ 1187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:01:27.649005 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:01:27.649006 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:01:27.649006 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:01:27.649049 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:01:27.649217 (+ 168us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:01:27.649217 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:01:27.649217 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:01:27.649304 (+ 87us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:01:27.649346 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0619 09:01:27.649534 (+ 188us) server_negotiation.cc:299] Negotiation successful
0619 09:01:27.649585 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260619 09:01:27.649803 6978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:27.641867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44403 (local address 127.0.98.3:55349)
0619 09:01:27.642053 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0619 09:01:27.642094 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 09:01:27.642623 (+ 529us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:01:27.642855 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:01:27.642858 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:01:27.643395 (+ 537us) client_negotiation.cc:189] Negotiated authn=SASL
0619 09:01:27.643572 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:27.643577 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:27.644180 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:27.644181 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:27.645250 (+ 1069us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:27.645253 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:27.645620 (+ 367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:27.645621 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:27.645722 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:27.646093 (+ 371us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 09:01:27.646100 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0619 09:01:27.647414 (+ 1314us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 09:01:27.648758 (+ 1344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:01:27.648763 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:01:27.648765 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:01:27.648953 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:01:27.649090 (+ 137us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:01:27.649090 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:01:27.649090 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:01:27.649175 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:01:27.649343 (+ 168us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 09:01:27.649345 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 09:01:27.649450 (+ 105us) client_negotiation.cc:769] Sending connection context
0619 09:01:27.649518 (+ 68us) client_negotiation.cc:240] Negotiation successful
0619 09:01:27.649623 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":79,"threads_started":1}
I20260619 09:01:27.650234 6976 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44403
I20260619 09:01:27.650317 6976 heartbeater.cc:464] Registering TS with master...
I20260619 09:01:27.650447 6976 heartbeater.cc:511] Master 127.0.98.62:44403 requested a full tablet report, sending...
I20260619 09:01:27.650756 6517 ts_manager.cc:194] Registered new tserver with Master: a5cc45b850424e7da92acf9089450d80 (127.0.98.3:34875)
I20260619 09:01:27.651117 6517 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:55349
I20260619 09:01:27.653388 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859658134510-392-0/krb5kdc/oryx_127.0.98.61.keytab.
Entry for principal oryx/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859658134510-392-0/krb5kdc/oryx_127.0.98.61.keytab.
I20260619 09:01:27.676841 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ez5NB/build/release/bin/kudu
/tmp/dist-test-task4ez5NB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859658134510-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859658134510-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859658134510-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859658134510-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:34401
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859658134510-392-0/krb5kdc/oryx_127.0.98.61.keytab
--principal=oryx/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42929
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44403,127.0.98.61:34401
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859658134510-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859658134510-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859658134510-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 09:01:27.754751 6981 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 09:01:27.754979 6981 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 09:01:27.755002 6981 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 09:01:27.756120 6981 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 09:01:27.756152 6981 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 09:01:27.756165 6981 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 09:01:27.756176 6981 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 09:01:27.756191 6981 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 09:01:27.757586 6981 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42929
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859658134510-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859658134510-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44403,127.0.98.61:34401
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859658134510-392-0/krb5kdc/oryx_127.0.98.61.keytab
--principal=oryx/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:34401
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859658134510-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6981
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859658134510-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
W20260619 09:01:27.757678 6981 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.0.98.62:44403,127.0.98.61:34401'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260619 09:01:27.757784 6981 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 09:01:27.757951 6981 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 09:01:27.762436 6981 server_base.cc:1034] running on GCE node
W20260619 09:01:27.762363 6987 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:01:27.762390 6989 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:01:27.762373 6986 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:01:27.762807 6981 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 09:01:27.763116 6981 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 09:01:27.764251 6981 hybrid_clock.cc:648] HybridClock initialized: now 1781859687764238 us; error 28 us; skew 500 ppm
Jun 19 09:01:27 dist-test-slave-mtdr krb5kdc[6472](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859687, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 09:01:27.766469 6981 init.cc:377] Logged in from keytab as oryx/127.0.98.61@KRBTEST.COM (short username oryx)
I20260619 09:01:27.766897 6981 webserver.cc:460] Webserver started at http://127.0.98.61:38097/ using document root <none> and password file <none>
I20260619 09:01:27.767103 6981 fs_manager.cc:359] Metadata directory not provided
I20260619 09:01:27.767151 6981 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 09:01:27.767238 6981 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 09:01:27.768187 6981 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859658134510-392-0/minicluster-data/master-1/data/instance:
uuid: "13ea0fb4e8094bd8bda6763a0a07c2c5"
format_stamp: "Formatted at 2026-06-19 09:01:27 on dist-test-slave-mtdr"
server_key: "a313548ca432c035f09151c1d6e5dca1"
server_key_iv: "1e67427340f68a35363279efdf63d9c3"
server_key_version: "encryptionkey@0"
I20260619 09:01:27.768455 6981 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859658134510-392-0/minicluster-data/master-1/wal/instance:
uuid: "13ea0fb4e8094bd8bda6763a0a07c2c5"
format_stamp: "Formatted at 2026-06-19 09:01:27 on dist-test-slave-mtdr"
server_key: "a313548ca432c035f09151c1d6e5dca1"
server_key_iv: "1e67427340f68a35363279efdf63d9c3"
server_key_version: "encryptionkey@0"
I20260619 09:01:27.769574 6981 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260619 09:01:27.770182 6996 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 09:01:27.770340 6981 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260619 09:01:27.770419 6981 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859658134510-392-0/minicluster-data/master-1/data,/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859658134510-392-0/minicluster-data/master-1/wal
uuid: "13ea0fb4e8094bd8bda6763a0a07c2c5"
format_stamp: "Formatted at 2026-06-19 09:01:27 on dist-test-slave-mtdr"
server_key: "a313548ca432c035f09151c1d6e5dca1"
server_key_iv: "1e67427340f68a35363279efdf63d9c3"
server_key_version: "encryptionkey@0"
I20260619 09:01:27.770475 6981 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859658134510-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859658134510-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859658134510-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 09:01:27.787293 6981 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 09:01:27.788882 6981 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 09:01:27.789005 6981 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 09:01:27.793354 6981 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:34401
I20260619 09:01:27.793380 7048 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:34401 every 8 connection(s)
I20260619 09:01:27.793727 6981 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859658134510-392-0/minicluster-data/master-1/data/info.pb
I20260619 09:01:27.794603 7049 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 09:01:27.796067 7049 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44403 } has no permanent_uuid. Determining permanent_uuid...
I20260619 09:01:27.800956 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ez5NB/build/release/bin/kudu as pid 6981
Jun 19 09:01:27 dist-test-slave-mtdr krb5kdc[6472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859687, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260619 09:01:27.804402 6576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:27.796801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43024 (local address 127.0.98.62:44403)
0619 09:01:27.796871 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:27.796873 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:27.796997 (+ 124us) server_negotiation.cc:407] Connection header received
0619 09:01:27.797427 (+ 430us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:27.797428 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:27.797438 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:27.797480 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:01:27.798099 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:27.798588 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:27.799610 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:27.799812 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:27.799863 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:27.801957 (+ 2094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:01:27.801958 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:01:27.801959 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:01:27.801965 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:01:27.803188 (+ 1223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:01:27.803531 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:01:27.803533 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:01:27.803533 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:01:27.803589 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:01:27.803797 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:01:27.803798 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:01:27.803799 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:01:27.803933 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:01:27.803988 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0619 09:01:27.804241 (+ 253us) server_negotiation.cc:299] Negotiation successful
0619 09:01:27.804315 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260619 09:01:27.804555 7051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:27.796669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44403 (local address 127.0.0.1:43024)
0619 09:01:27.796851 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0619 09:01:27.796894 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 09:01:27.797335 (+ 441us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:01:27.797554 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:01:27.797557 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:01:27.797830 (+ 273us) client_negotiation.cc:189] Negotiated authn=SASL
0619 09:01:27.798014 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:27.798018 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:27.798663 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:27.798664 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:27.799529 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:27.799532 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:27.799895 (+ 363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:27.799896 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:27.800015 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:27.800436 (+ 421us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 09:01:27.800445 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0619 09:01:27.801879 (+ 1434us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 09:01:27.803257 (+ 1378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:01:27.803262 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:01:27.803265 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:01:27.803467 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:01:27.803641 (+ 174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:01:27.803642 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:01:27.803643 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:01:27.803744 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:01:27.803989 (+ 245us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 09:01:27.803994 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 09:01:27.804129 (+ 135us) client_negotiation.cc:769] Sending connection context
0619 09:01:27.804214 (+ 85us) client_negotiation.cc:240] Negotiation successful
0619 09:01:27.804351 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":60,"threads_started":1}
I20260619 09:01:27.805091 7049 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34401 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 09:01:27 dist-test-slave-mtdr krb5kdc[6472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859687, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260619 09:01:27.808779 7053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:27.801339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43210 (local address 127.0.98.61:34401)
0619 09:01:27.801478 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:27.801483 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:27.801504 (+ 21us) server_negotiation.cc:407] Connection header received
0619 09:01:27.801524 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:27.801526 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:27.801544 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:27.801613 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:01:27.802197 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:27.803062 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:27.803737 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:27.804021 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:27.804076 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:27.806361 (+ 2285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:01:27.806369 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:01:27.806369 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:01:27.806376 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:01:27.807728 (+ 1352us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:01:27.808037 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:01:27.808040 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:01:27.808041 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:01:27.808096 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:01:27.808291 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:01:27.808292 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:01:27.808292 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:01:27.808421 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:01:27.808537 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0619 09:01:27.808631 (+ 94us) server_negotiation.cc:299] Negotiation successful
0619 09:01:27.808688 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":74,"threads_started":1}
Jun 19 09:01:27 dist-test-slave-mtdr krb5kdc[6472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859687, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260619 09:01:27.812048 7051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:27.805292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34401 (local address 127.0.0.1:43212)
0619 09:01:27.805453 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0619 09:01:27.805462 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 09:01:27.805506 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:01:27.806058 (+ 552us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:01:27.806059 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:01:27.806188 (+ 129us) client_negotiation.cc:189] Negotiated authn=SASL
0619 09:01:27.806237 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:27.806238 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:27.807112 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:27.807113 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:27.807568 (+ 455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:27.807569 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:27.807892 (+ 323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:27.807893 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:27.807952 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:27.808245 (+ 293us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 09:01:27.808249 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 09:01:27.809410 (+ 1161us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 09:01:27.811240 (+ 1830us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:01:27.811241 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:01:27.811241 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:01:27.811378 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:01:27.811528 (+ 150us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:01:27.811529 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:01:27.811530 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:01:27.811577 (+ 47us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:01:27.811775 (+ 198us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 09:01:27.811776 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 09:01:27.811809 (+ 33us) client_negotiation.cc:769] Sending connection context
0619 09:01:27.811868 (+ 59us) client_negotiation.cc:240] Negotiation successful
0619 09:01:27.811953 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147}
I20260619 09:01:27.812333 7054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:27.805341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43212 (local address 127.0.98.61:34401)
0619 09:01:27.805466 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:27.805468 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:27.805954 (+ 486us) server_negotiation.cc:407] Connection header received
0619 09:01:27.805982 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:27.805983 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:27.805991 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:27.806042 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:01:27.806327 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:27.807032 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:27.807638 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:27.807840 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:27.807890 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:27.809477 (+ 1587us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:01:27.809478 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:01:27.809479 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:01:27.809485 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:01:27.811175 (+ 1690us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:01:27.811433 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:01:27.811434 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:01:27.811434 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:01:27.811482 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:01:27.811632 (+ 150us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:01:27.811633 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:01:27.811634 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:01:27.811726 (+ 92us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:01:27.811777 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0619 09:01:27.811891 (+ 114us) server_negotiation.cc:299] Negotiation successful
0619 09:01:27.812001 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":52,"server-negotiator.queue_time_us":108,"thread_start_us":80,"threads_started":1}
I20260619 09:01:27.813871 7049 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 13ea0fb4e8094bd8bda6763a0a07c2c5: Bootstrap starting.
I20260619 09:01:27.814604 7049 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 13ea0fb4e8094bd8bda6763a0a07c2c5: Neither blocks nor log segments found. Creating new log.
I20260619 09:01:27.814899 7049 log.cc:824] T 00000000000000000000000000000000 P 13ea0fb4e8094bd8bda6763a0a07c2c5: Log is configured to *not* fsync() on all Append() calls
I20260619 09:01:27.815541 7049 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 13ea0fb4e8094bd8bda6763a0a07c2c5: No bootstrap required, opened a new log
I20260619 09:01:27.816845 7049 raft_consensus.cc:348] T 00000000000000000000000000000000 P 13ea0fb4e8094bd8bda6763a0a07c2c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aaa8435622414a159f995f651c058e08" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44403 } } peers { permanent_uuid: "13ea0fb4e8094bd8bda6763a0a07c2c5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34401 } }
I20260619 09:01:27.816962 7049 raft_consensus.cc:374] T 00000000000000000000000000000000 P 13ea0fb4e8094bd8bda6763a0a07c2c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 09:01:27.816982 7049 raft_consensus.cc:729] T 00000000000000000000000000000000 P 13ea0fb4e8094bd8bda6763a0a07c2c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13ea0fb4e8094bd8bda6763a0a07c2c5, State: Initialized, Role: FOLLOWER
I20260619 09:01:27.817101 7049 consensus_queue.cc:260] T 00000000000000000000000000000000 P 13ea0fb4e8094bd8bda6763a0a07c2c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aaa8435622414a159f995f651c058e08" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44403 } } peers { permanent_uuid: "13ea0fb4e8094bd8bda6763a0a07c2c5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34401 } }
I20260619 09:01:27.817351 7056 sys_catalog.cc:455] T 00000000000000000000000000000000 P 13ea0fb4e8094bd8bda6763a0a07c2c5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aaa8435622414a159f995f651c058e08" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44403 } } peers { permanent_uuid: "13ea0fb4e8094bd8bda6763a0a07c2c5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34401 } } }
I20260619 09:01:27.817407 7049 sys_catalog.cc:564] T 00000000000000000000000000000000 P 13ea0fb4e8094bd8bda6763a0a07c2c5 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 09:01:27.817432 7056 sys_catalog.cc:458] T 00000000000000000000000000000000 P 13ea0fb4e8094bd8bda6763a0a07c2c5 [sys.catalog]: This master's current role is: FOLLOWER
W20260619 09:01:27.818930 7067 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 13ea0fb4e8094bd8bda6763a0a07c2c5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 09:01:27.819024 7067 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 09:01:27.820520 6981 master_runner.cc:428] Detected that this master 13ea0fb4e8094bd8bda6763a0a07c2c5 is joining an existing cluster
I20260619 09:01:27.820557 6981 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.98.61:34401
I20260619 09:01:27.827427 7054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:27.821673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43220 (local address 127.0.98.61:34401)
0619 09:01:27.821710 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:27.821712 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:27.821905 (+ 193us) server_negotiation.cc:407] Connection header received
0619 09:01:27.821938 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:27.821939 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:27.821950 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:27.822018 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:01:27.822265 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:27.822758 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:27.823837 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:27.824045 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:27.824094 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:27.825013 (+ 919us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:01:27.825014 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:01:27.825015 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:01:27.825020 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:01:27.826344 (+ 1324us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:01:27.826608 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:01:27.826609 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:01:27.826610 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:01:27.826655 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:01:27.826802 (+ 147us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:01:27.826802 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:01:27.826803 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:01:27.826928 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:01:27.826991 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0619 09:01:27.827185 (+ 194us) server_negotiation.cc:299] Negotiation successful
0619 09:01:27.827320 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":63,"server-negotiator.queue_time_us":21}
I20260619 09:01:27.827605 7074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:27.821727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34401 (local address 127.0.0.1:43220)
0619 09:01:27.821841 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0619 09:01:27.821849 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0619 09:01:27.821899 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:01:27.822024 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:01:27.822024 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:01:27.822166 (+ 142us) client_negotiation.cc:189] Negotiated authn=SASL
0619 09:01:27.822204 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:27.822206 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:27.822847 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:27.822848 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:27.823743 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:27.823745 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:27.824099 (+ 354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:27.824100 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:27.824169 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:27.824557 (+ 388us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 09:01:27.824561 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 09:01:27.824945 (+ 384us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 09:01:27.826408 (+ 1463us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:01:27.826408 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:01:27.826409 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:01:27.826554 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:01:27.826704 (+ 150us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:01:27.826705 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:01:27.826705 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:01:27.826755 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:01:27.826989 (+ 234us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 09:01:27.826990 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 09:01:27.827046 (+ 56us) client_negotiation.cc:769] Sending connection context
0619 09:01:27.827159 (+ 113us) client_negotiation.cc:240] Negotiation successful
0619 09:01:27.827263 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":69,"threads_started":1}
I20260619 09:01:27.828054 6576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:27.821413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43026 (local address 127.0.98.62:44403)
0619 09:01:27.821450 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:27.821451 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:27.821597 (+ 146us) server_negotiation.cc:407] Connection header received
0619 09:01:27.821682 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:27.821683 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:27.821692 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:27.821735 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:01:27.822107 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:27.822806 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:27.823825 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:27.824109 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:27.824200 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:27.825276 (+ 1076us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:01:27.825278 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:01:27.825278 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:01:27.825285 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:01:27.826784 (+ 1499us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:01:27.827134 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:01:27.827135 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:01:27.827135 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:01:27.827187 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:01:27.827381 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:01:27.827382 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:01:27.827382 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:01:27.827498 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:01:27.827541 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0619 09:01:27.827897 (+ 356us) server_negotiation.cc:299] Negotiation successful
0619 09:01:27.827968 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260619 09:01:27.828328 7073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:27.821381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44403 (local address 127.0.0.1:43026)
0619 09:01:27.821538 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0619 09:01:27.821547 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 09:01:27.821599 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:01:27.821744 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:01:27.821745 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:01:27.821931 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0619 09:01:27.822006 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:27.822008 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:27.822931 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:27.822931 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:27.823745 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:27.823746 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:27.824282 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:27.824283 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:27.824349 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:27.824705 (+ 356us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 09:01:27.824709 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 09:01:27.825165 (+ 456us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 09:01:27.826905 (+ 1740us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:01:27.826905 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:01:27.826906 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:01:27.827076 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:01:27.827251 (+ 175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:01:27.827252 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:01:27.827252 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:01:27.827301 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:01:27.827749 (+ 448us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 09:01:27.827750 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 09:01:27.827791 (+ 41us) client_negotiation.cc:769] Sending connection context
0619 09:01:27.828130 (+ 339us) client_negotiation.cc:240] Negotiation successful
0619 09:01:27.828223 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":95,"threads_started":1}
Jun 19 09:01:27 dist-test-slave-mtdr krb5kdc[6472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859687, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260619 09:01:27.837209 7075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:27.830263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34401 (local address 127.0.0.1:43234)
0619 09:01:27.830365 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0619 09:01:27.830404 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 09:01:27.830466 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:01:27.830602 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:01:27.830604 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:01:27.830892 (+ 288us) client_negotiation.cc:189] Negotiated authn=SASL
0619 09:01:27.830945 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:27.830947 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:27.831812 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:27.831812 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:27.832411 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:27.832413 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:27.832863 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:27.832864 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:27.832918 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:27.833280 (+ 362us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 09:01:27.833283 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 09:01:27.834754 (+ 1471us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 09:01:27.836311 (+ 1557us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:01:27.836325 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:01:27.836327 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:01:27.836503 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:01:27.836687 (+ 184us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:01:27.836687 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:01:27.836687 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:01:27.836733 (+ 46us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:01:27.836958 (+ 225us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 09:01:27.836961 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 09:01:27.837000 (+ 39us) client_negotiation.cc:769] Sending connection context
0619 09:01:27.837057 (+ 57us) client_negotiation.cc:240] Negotiation successful
0619 09:01:27.837138 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":79,"thread_start_us":50,"threads_started":1}
I20260619 09:01:27.837260 7054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:27.830319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43234 (local address 127.0.98.61:34401)
0619 09:01:27.830358 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:27.830359 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:27.830462 (+ 103us) server_negotiation.cc:407] Connection header received
0619 09:01:27.830518 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:27.830519 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:27.830529 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:27.830600 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:01:27.831012 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:27.831739 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:27.832482 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:27.832765 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:27.832863 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:27.834864 (+ 2001us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:01:27.834865 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:01:27.834865 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:01:27.834872 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:01:27.836232 (+ 1360us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:01:27.836567 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:01:27.836568 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:01:27.836568 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:01:27.836627 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:01:27.836779 (+ 152us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:01:27.836779 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:01:27.836780 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:01:27.836902 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:01:27.836971 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0619 09:01:27.837101 (+ 130us) server_negotiation.cc:299] Negotiation successful
0619 09:01:27.837172 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260619 09:01:27.837724 6517 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.98.61:34401: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "13ea0fb4e8094bd8bda6763a0a07c2c5"
member_type: NON_VOTER
last_known_addr {
host: "127.0.98.61"
port: 34401
}
attrs {
promote: true
}
}
dest_uuid: "aaa8435622414a159f995f651c058e08"
cas_config_opid_index: -1
I20260619 09:01:27.838048 6517 consensus_queue.cc:237] T 00000000000000000000000000000000 P aaa8435622414a159f995f651c058e08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "aaa8435622414a159f995f651c058e08" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44403 } } peers { permanent_uuid: "13ea0fb4e8094bd8bda6763a0a07c2c5" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 34401 } attrs { promote: true } }
I20260619 09:01:27.838529 7077 raft_consensus.cc:2932] T 00000000000000000000000000000000 P aaa8435622414a159f995f651c058e08 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 13ea0fb4e8094bd8bda6763a0a07c2c5 (127.0.98.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "aaa8435622414a159f995f651c058e08" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44403 } } peers { permanent_uuid: "13ea0fb4e8094bd8bda6763a0a07c2c5" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 34401 } attrs { promote: true } } }
I20260619 09:01:27.838788 7023 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 13ea0fb4e8094bd8bda6763a0a07c2c5 [term 0 FOLLOWER]: Advancing to term 1
I20260619 09:01:27.839485 7023 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 13ea0fb4e8094bd8bda6763a0a07c2c5 [term 1 FOLLOWER]: Refusing update from remote peer aaa8435622414a159f995f651c058e08: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260619 09:01:27.839550 7077 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.98.61:34401
I20260619 09:01:27.839761 6981 master.cc:537] Master@127.0.98.61:34401 shutting down...
I20260619 09:01:27.839699 7076 sys_catalog.cc:455] T 00000000000000000000000000000000 P aaa8435622414a159f995f651c058e08 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "aaa8435622414a159f995f651c058e08" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "aaa8435622414a159f995f651c058e08" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44403 } } peers { permanent_uuid: "13ea0fb4e8094bd8bda6763a0a07c2c5" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 34401 } attrs { promote: true } } }
I20260619 09:01:27.840188 7076 sys_catalog.cc:458] T 00000000000000000000000000000000 P aaa8435622414a159f995f651c058e08 [sys.catalog]: This master's current role is: LEADER
I20260619 09:01:27.839794 7077 consensus_queue.cc:1035] T 00000000000000000000000000000000 P aaa8435622414a159f995f651c058e08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "13ea0fb4e8094bd8bda6763a0a07c2c5" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 34401 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260619 09:01:27.841503 6504 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260619 09:01:27.841902 6504 consensus_peers.cc:487] T 00000000000000000000000000000000 P aaa8435622414a159f995f651c058e08 -> Peer 13ea0fb4e8094bd8bda6763a0a07c2c5 (127.0.98.61:34401): Couldn't send request to peer 13ea0fb4e8094bd8bda6763a0a07c2c5. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260619 09:01:27.842361 6981 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 13ea0fb4e8094bd8bda6763a0a07c2c5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 09:01:27.842430 6981 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 13ea0fb4e8094bd8bda6763a0a07c2c5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 09:01:27.842476 6981 tablet_replica.cc:330] T 00000000000000000000000000000000 P 13ea0fb4e8094bd8bda6763a0a07c2c5: stopping tablet replica
I20260619 09:01:27.855008 6981 master.cc:552] Master@127.0.98.61:34401 shutdown complete.
I20260619 09:01:27.855067 6981 master_runner.cc:305] Clearing existing system tablet
I20260619 09:01:27.855202 6981 fs_manager.cc:359] Metadata directory not provided
I20260619 09:01:27.855247 6981 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 09:01:27.855914 6981 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.001s
I20260619 09:01:27.856372 7081 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 09:01:27.856487 6981 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859658134510-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 09:01:27.856541 6981 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260619 09:01:27.856576 6981 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859658134510-392-0/minicluster-data/master-1/data,/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859658134510-392-0/minicluster-data/master-1/wal
uuid: "13ea0fb4e8094bd8bda6763a0a07c2c5"
format_stamp: "Formatted at 2026-06-19 09:01:27 on dist-test-slave-mtdr"
server_key: "a313548ca432c035f09151c1d6e5dca1"
server_key_iv: "1e67427340f68a35363279efdf63d9c3"
server_key_version: "encryptionkey@0"
I20260619 09:01:27.857354 6981 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 13ea0fb4e8094bd8bda6763a0a07c2c5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 09:01:27.859112 6981 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 13ea0fb4e8094bd8bda6763a0a07c2c5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260619 09:01:27.859194 6981 log.cc:1196] T 00000000000000000000000000000000 P 13ea0fb4e8094bd8bda6763a0a07c2c5: Deleting WAL directory at /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859658134510-392-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260619 09:01:27.859411 6981 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 13ea0fb4e8094bd8bda6763a0a07c2c5: Deleting consensus metadata
I20260619 09:01:27.859532 6981 master_runner.cc:315] Copying system tablet from 127.0.98.62:44403
I20260619 09:01:27.860201 6981 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 13ea0fb4e8094bd8bda6763a0a07c2c5: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.62:44403
I20260619 09:01:27.866091 6576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:27.860533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43036 (local address 127.0.98.62:44403)
0619 09:01:27.860583 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:27.860585 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:27.860708 (+ 123us) server_negotiation.cc:407] Connection header received
0619 09:01:27.860776 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:27.860776 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:27.860785 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:27.860833 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:01:27.861192 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:27.861671 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:27.862733 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:27.862990 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:27.863054 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:27.863928 (+ 874us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:01:27.863930 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:01:27.863930 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:01:27.863937 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:01:27.865140 (+ 1203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:01:27.865429 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:01:27.865430 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:01:27.865431 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:01:27.865483 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:01:27.865636 (+ 153us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:01:27.865636 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:01:27.865637 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:01:27.865741 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:01:27.865794 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0619 09:01:27.865931 (+ 137us) server_negotiation.cc:299] Negotiation successful
0619 09:01:27.865997 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260619 09:01:27.866107 7087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:27.860477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44403 (local address 127.0.0.1:43036)
0619 09:01:27.860620 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0619 09:01:27.860633 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 09:01:27.860717 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:01:27.860843 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:01:27.860844 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:01:27.861044 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0619 09:01:27.861112 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:27.861114 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:27.861735 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:27.861736 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:27.862650 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:27.862653 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:27.863056 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:27.863057 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:27.863150 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:27.863562 (+ 412us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 09:01:27.863567 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 09:01:27.863848 (+ 281us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 09:01:27.865212 (+ 1364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:01:27.865213 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:01:27.865213 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:01:27.865371 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:01:27.865533 (+ 162us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:01:27.865533 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:01:27.865534 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:01:27.865587 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:01:27.865791 (+ 204us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 09:01:27.865792 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 09:01:27.865844 (+ 52us) client_negotiation.cc:769] Sending connection context
0619 09:01:27.865902 (+ 58us) client_negotiation.cc:240] Negotiation successful
0619 09:01:27.865999 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":81,"threads_started":1}
I20260619 09:01:27.866525 6537 tablet_copy_service.cc:140] P aaa8435622414a159f995f651c058e08: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 13ea0fb4e8094bd8bda6763a0a07c2c5 ({username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:43036)
I20260619 09:01:27.866602 6537 tablet_copy_service.cc:161] P aaa8435622414a159f995f651c058e08: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 13ea0fb4e8094bd8bda6763a0a07c2c5 at {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:43036: session id = 13ea0fb4e8094bd8bda6763a0a07c2c5-00000000000000000000000000000000
I20260619 09:01:27.867348 6537 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P aaa8435622414a159f995f651c058e08: Tablet Copy: opened 0 blocks and 1 log segments
I20260619 09:01:27.867918 6981 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260619 09:01:27.869122 6981 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 13ea0fb4e8094bd8bda6763a0a07c2c5: tablet copy: Starting download of 0 data blocks...
I20260619 09:01:27.869277 6981 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 13ea0fb4e8094bd8bda6763a0a07c2c5: tablet copy: Starting download of 1 WAL segments...
I20260619 09:01:27.870059 6981 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 13ea0fb4e8094bd8bda6763a0a07c2c5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260619 09:01:27.870949 6537 tablet_copy_service.cc:342] P aaa8435622414a159f995f651c058e08: Request end of tablet copy session 13ea0fb4e8094bd8bda6763a0a07c2c5-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:43036
I20260619 09:01:27.870995 6537 tablet_copy_service.cc:433] P aaa8435622414a159f995f651c058e08: ending tablet copy session 13ea0fb4e8094bd8bda6763a0a07c2c5-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 13ea0fb4e8094bd8bda6763a0a07c2c5
I20260619 09:01:27.872960 6981 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 09:01:27.873922 7088 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:01:27.874033 7089 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:01:27.874190 7091 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:01:27.874311 6981 server_base.cc:1034] running on GCE node
I20260619 09:01:27.874414 6981 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 09:01:27.874536 6981 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 09:01:27.875653 6981 hybrid_clock.cc:648] HybridClock initialized: now 1781859687875642 us; error 21 us; skew 500 ppm
Jun 19 09:01:27 dist-test-slave-mtdr krb5kdc[6472](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859687, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 09:01:27.876914 6981 init.cc:377] Logged in from keytab as oryx/127.0.98.61@KRBTEST.COM (short username oryx)
I20260619 09:01:27.877166 6981 webserver.cc:460] Webserver started at http://127.0.98.61:34857/ using document root <none> and password file <none>
I20260619 09:01:27.877283 6981 fs_manager.cc:359] Metadata directory not provided
I20260619 09:01:27.877316 6981 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 09:01:27.877943 6981 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.000s
I20260619 09:01:27.878460 7098 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 09:01:27.878563 6981 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260619 09:01:27.878603 6981 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859658134510-392-0/minicluster-data/master-1/data,/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859658134510-392-0/minicluster-data/master-1/wal
uuid: "13ea0fb4e8094bd8bda6763a0a07c2c5"
format_stamp: "Formatted at 2026-06-19 09:01:27 on dist-test-slave-mtdr"
server_key: "a313548ca432c035f09151c1d6e5dca1"
server_key_iv: "1e67427340f68a35363279efdf63d9c3"
server_key_version: "encryptionkey@0"
I20260619 09:01:27.878650 6981 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859658134510-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859658134510-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859658134510-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 09:01:27.892815 6981 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 09:01:27.894112 6981 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 09:01:27.896492 6981 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:34401
I20260619 09:01:27.896570 7150 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:34401 every 8 connection(s)
I20260619 09:01:27.896775 6981 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859658134510-392-0/minicluster-data/master-1/data/info.pb
I20260619 09:01:27.897360 7151 sys_catalog.cc:263] Verifying existing consensus state
I20260619 09:01:27.897735 7151 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 13ea0fb4e8094bd8bda6763a0a07c2c5: Bootstrap starting.
I20260619 09:01:27.899809 7151 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 13ea0fb4e8094bd8bda6763a0a07c2c5: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 09:01:27.900015 7151 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 13ea0fb4e8094bd8bda6763a0a07c2c5: Bootstrap complete.
I20260619 09:01:27.900180 7151 raft_consensus.cc:348] T 00000000000000000000000000000000 P 13ea0fb4e8094bd8bda6763a0a07c2c5 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "aaa8435622414a159f995f651c058e08" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44403 } } peers { permanent_uuid: "13ea0fb4e8094bd8bda6763a0a07c2c5" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 34401 } attrs { promote: true } }
I20260619 09:01:27.900245 7151 raft_consensus.cc:729] T 00000000000000000000000000000000 P 13ea0fb4e8094bd8bda6763a0a07c2c5 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 13ea0fb4e8094bd8bda6763a0a07c2c5, State: Initialized, Role: LEARNER
I20260619 09:01:27.900269 7151 consensus_queue.cc:260] T 00000000000000000000000000000000 P 13ea0fb4e8094bd8bda6763a0a07c2c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "aaa8435622414a159f995f651c058e08" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44403 } } peers { permanent_uuid: "13ea0fb4e8094bd8bda6763a0a07c2c5" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 34401 } attrs { promote: true } }
I20260619 09:01:27.900421 7151 sys_catalog.cc:564] T 00000000000000000000000000000000 P 13ea0fb4e8094bd8bda6763a0a07c2c5 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 09:01:27.900416 7154 sys_catalog.cc:455] T 00000000000000000000000000000000 P 13ea0fb4e8094bd8bda6763a0a07c2c5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "aaa8435622414a159f995f651c058e08" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44403 } } peers { permanent_uuid: "13ea0fb4e8094bd8bda6763a0a07c2c5" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 34401 } attrs { promote: true } } }
I20260619 09:01:27.900525 7154 sys_catalog.cc:458] T 00000000000000000000000000000000 P 13ea0fb4e8094bd8bda6763a0a07c2c5 [sys.catalog]: This master's current role is: LEARNER
I20260619 09:01:27.901630 7165 catalog_manager.cc:1259] Loaded cluster ID: aac0526a83f348348927f89c7ec729d5
I20260619 09:01:27.902117 7165 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 13ea0fb4e8094bd8bda6763a0a07c2c5: loading cluster ID for follower catalog manager: success
I20260619 09:01:27.903244 7165 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 13ea0fb4e8094bd8bda6763a0a07c2c5: acquiring CA information for follower catalog manager: success
I20260619 09:01:27.903544 7165 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 13ea0fb4e8094bd8bda6763a0a07c2c5: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 09:01:27.920377 7166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:27.913543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43246 (local address 127.0.98.61:34401)
0619 09:01:27.913672 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:27.913674 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:27.913684 (+ 10us) server_negotiation.cc:407] Connection header received
0619 09:01:27.913712 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:27.913713 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:27.913724 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:27.913800 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:01:27.914354 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:27.915444 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:27.916170 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:27.916498 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:27.916568 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:27.917668 (+ 1100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:01:27.917669 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:01:27.917669 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:01:27.917677 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:01:27.919277 (+ 1600us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:01:27.919579 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:01:27.919580 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:01:27.919580 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:01:27.919644 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:01:27.919785 (+ 141us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:01:27.919785 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:01:27.919786 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:01:27.919912 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:01:27.920008 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0619 09:01:27.920079 (+ 71us) server_negotiation.cc:299] Negotiation successful
0619 09:01:27.920156 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":63,"threads_started":1}
WARNING: no policy specified for oryx/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.60@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859658134510-392-0/krb5kdc/oryx_127.0.98.60.keytab.
Entry for principal oryx/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859658134510-392-0/krb5kdc/oryx_127.0.98.60.keytab.
I20260619 09:01:27.942677 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ez5NB/build/release/bin/kudu
/tmp/dist-test-task4ez5NB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859658134510-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859658134510-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859658134510-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859658134510-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:39539
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859658134510-392-0/krb5kdc/oryx_127.0.98.60.keytab
--principal=oryx/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42929
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44403,127.0.98.61:34401,127.0.98.60:39539
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859658134510-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859658134510-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859658134510-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 09:01:28.021631 7169 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 09:01:28.021838 7169 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 09:01:28.021859 7169 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 09:01:28.023051 7169 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 09:01:28.023101 7169 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 09:01:28.023116 7169 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 09:01:28.023128 7169 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 09:01:28.023139 7169 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 09:01:28.024652 7169 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42929
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859658134510-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859658134510-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44403,127.0.98.61:34401,127.0.98.60:39539
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859658134510-392-0/krb5kdc/oryx_127.0.98.60.keytab
--principal=oryx/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:39539
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859658134510-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7169
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859658134510-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 09:01:28.024883 7169 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 09:01:28.025092 7169 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 09:01:28.029549 7177 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:01:28.029666 7169 server_base.cc:1034] running on GCE node
W20260619 09:01:28.029541 7175 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:01:28.029536 7174 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:01:28.029954 7169 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 09:01:28.030215 7169 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 09:01:28.031345 7169 hybrid_clock.cc:648] HybridClock initialized: now 1781859688031321 us; error 37 us; skew 500 ppm
Jun 19 09:01:28 dist-test-slave-mtdr krb5kdc[6472](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859688, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 09:01:28.033675 7169 init.cc:377] Logged in from keytab as oryx/127.0.98.60@KRBTEST.COM (short username oryx)
I20260619 09:01:28.034082 7169 webserver.cc:460] Webserver started at http://127.0.98.60:33053/ using document root <none> and password file <none>
I20260619 09:01:28.034272 7169 fs_manager.cc:359] Metadata directory not provided
I20260619 09:01:28.034319 7169 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 09:01:28.034435 7169 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 09:01:28.035486 7169 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859658134510-392-0/minicluster-data/master-2/data/instance:
uuid: "f1435fb0f12c416196a176e829751c3b"
format_stamp: "Formatted at 2026-06-19 09:01:28 on dist-test-slave-mtdr"
server_key: "c91f639a49b8dea10b67aa055d0638b7"
server_key_iv: "a309b464eb8d861c61a731fee2058a98"
server_key_version: "encryptionkey@0"
I20260619 09:01:28.035804 7169 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859658134510-392-0/minicluster-data/master-2/wal/instance:
uuid: "f1435fb0f12c416196a176e829751c3b"
format_stamp: "Formatted at 2026-06-19 09:01:28 on dist-test-slave-mtdr"
server_key: "c91f639a49b8dea10b67aa055d0638b7"
server_key_iv: "a309b464eb8d861c61a731fee2058a98"
server_key_version: "encryptionkey@0"
I20260619 09:01:28.037024 7169 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 09:01:28.037644 7184 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 09:01:28.037827 7169 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 09:01:28.037916 7169 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859658134510-392-0/minicluster-data/master-2/data,/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859658134510-392-0/minicluster-data/master-2/wal
uuid: "f1435fb0f12c416196a176e829751c3b"
format_stamp: "Formatted at 2026-06-19 09:01:28 on dist-test-slave-mtdr"
server_key: "c91f639a49b8dea10b67aa055d0638b7"
server_key_iv: "a309b464eb8d861c61a731fee2058a98"
server_key_version: "encryptionkey@0"
I20260619 09:01:28.037981 7169 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859658134510-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859658134510-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859658134510-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 09:01:28.047917 7169 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 09:01:28.049997 7169 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 09:01:28.050128 7169 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 09:01:28.053867 7169 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:39539
I20260619 09:01:28.053886 7236 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:39539 every 8 connection(s)
I20260619 09:01:28.054279 7169 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859658134510-392-0/minicluster-data/master-2/data/info.pb
I20260619 09:01:28.055249 7237 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 09:01:28.057096 7237 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44403 } has no permanent_uuid. Determining permanent_uuid...
I20260619 09:01:28.057382 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ez5NB/build/release/bin/kudu as pid 7169
Jun 19 09:01:28 dist-test-slave-mtdr krb5kdc[6472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859688, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Jun 19 09:01:28 dist-test-slave-mtdr krb5kdc[6472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859687, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260619 09:01:28.065907 6576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:28.057888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43044 (local address 127.0.98.62:44403)
0619 09:01:28.058011 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:28.058013 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:28.058105 (+ 92us) server_negotiation.cc:407] Connection header received
0619 09:01:28.058571 (+ 466us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:28.058572 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:28.058583 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:28.058634 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:01:28.059194 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:28.059673 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:28.060825 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:28.061071 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:28.061134 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:28.063318 (+ 2184us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:01:28.063319 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:01:28.063320 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:01:28.063327 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:01:28.064581 (+ 1254us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:01:28.064995 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:01:28.064996 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:01:28.064997 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:01:28.065056 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:01:28.065258 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:01:28.065259 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:01:28.065259 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:01:28.065396 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:01:28.065467 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0619 09:01:28.065723 (+ 256us) server_negotiation.cc:299] Negotiation successful
0619 09:01:28.065790 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260619 09:01:28.066080 7239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:28.057750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44403 (local address 127.0.0.1:43044)
0619 09:01:28.057973 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0619 09:01:28.058017 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 09:01:28.058496 (+ 479us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:01:28.058653 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:01:28.058656 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:01:28.058953 (+ 297us) client_negotiation.cc:189] Negotiated authn=SASL
0619 09:01:28.059112 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:28.059116 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:28.059765 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:28.059766 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:28.060740 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:28.060742 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:28.061150 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:28.061151 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:28.061256 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:28.061695 (+ 439us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 09:01:28.061706 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 09:01:28.063216 (+ 1510us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 09:01:28.064644 (+ 1428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:01:28.064651 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:01:28.064653 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:01:28.064926 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:01:28.065119 (+ 193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:01:28.065119 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:01:28.065119 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:01:28.065205 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:01:28.065454 (+ 249us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 09:01:28.065457 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 09:01:28.065592 (+ 135us) client_negotiation.cc:769] Sending connection context
0619 09:01:28.065703 (+ 111us) client_negotiation.cc:240] Negotiation successful
0619 09:01:28.065868 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":88,"threads_started":1}
I20260619 09:01:28.066568 7237 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34401 } has no permanent_uuid. Determining permanent_uuid...
I20260619 09:01:28.066672 7240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:28.057896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34862 (local address 127.0.98.60:39539)
0619 09:01:28.058062 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:28.058066 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:28.058088 (+ 22us) server_negotiation.cc:407] Connection header received
0619 09:01:28.058485 (+ 397us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:28.058491 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:28.058510 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:28.058589 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:01:28.059209 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:28.060005 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:28.060934 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:28.061328 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:28.061377 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:28.063780 (+ 2403us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:01:28.063802 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:01:28.063803 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:01:28.063808 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:01:28.065472 (+ 1664us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:01:28.065813 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:01:28.065816 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:01:28.065816 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:01:28.065875 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:01:28.066076 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:01:28.066077 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:01:28.066078 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:01:28.066254 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:01:28.066425 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0619 09:01:28.066471 (+ 46us) server_negotiation.cc:299] Negotiation successful
0619 09:01:28.066540 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":80,"threads_started":1}
Jun 19 09:01:28 dist-test-slave-mtdr krb5kdc[6472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859688, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260619 09:01:28.072438 7239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:28.066846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34401 (local address 127.0.0.1:43258)
0619 09:01:28.066884 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0619 09:01:28.066892 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0619 09:01:28.066923 (+ 31us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:01:28.067078 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:01:28.067079 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:01:28.067209 (+ 130us) client_negotiation.cc:189] Negotiated authn=SASL
0619 09:01:28.067271 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:28.067273 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:28.067893 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:28.067894 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:28.068392 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:28.068394 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:28.068687 (+ 293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:28.068688 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:28.068734 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:28.069026 (+ 292us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 09:01:28.069030 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 09:01:28.070156 (+ 1126us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 09:01:28.071570 (+ 1414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:01:28.071571 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:01:28.071572 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:01:28.071724 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:01:28.071949 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:01:28.071950 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:01:28.071950 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:01:28.072012 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:01:28.072211 (+ 199us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 09:01:28.072211 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 09:01:28.072248 (+ 37us) client_negotiation.cc:769] Sending connection context
0619 09:01:28.072307 (+ 59us) client_negotiation.cc:240] Negotiation successful
0619 09:01:28.072376 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26}
I20260619 09:01:28.072454 7166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:28.066904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43258 (local address 127.0.98.61:34401)
0619 09:01:28.066940 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:28.066943 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:28.066951 (+ 8us) server_negotiation.cc:407] Connection header received
0619 09:01:28.066976 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:28.066977 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:28.066988 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:28.067046 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:01:28.067355 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:28.067835 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:28.068456 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:28.068637 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:28.068682 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:28.070249 (+ 1567us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:01:28.070250 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:01:28.070251 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:01:28.070256 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:01:28.071501 (+ 1245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:01:28.071806 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:01:28.071807 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:01:28.071807 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:01:28.071865 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:01:28.072070 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:01:28.072071 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:01:28.072071 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:01:28.072165 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:01:28.072209 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0619 09:01:28.072326 (+ 117us) server_negotiation.cc:299] Negotiation successful
0619 09:01:28.072388 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260619 09:01:28.072815 7237 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39539 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 09:01:28 dist-test-slave-mtdr krb5kdc[6472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859688, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260619 09:01:28.078231 7239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:28.073074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39539 (local address 127.0.0.1:34868)
0619 09:01:28.073118 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0619 09:01:28.073129 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 09:01:28.073172 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:01:28.073264 (+ 92us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:01:28.073265 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:01:28.073366 (+ 101us) client_negotiation.cc:189] Negotiated authn=SASL
0619 09:01:28.073394 (+ 28us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:28.073395 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:28.073935 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:28.073936 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:28.074388 (+ 452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:28.074390 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:28.074764 (+ 374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:28.074764 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:28.074801 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:28.075133 (+ 332us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 09:01:28.075137 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 09:01:28.076192 (+ 1055us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 09:01:28.077416 (+ 1224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:01:28.077417 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:01:28.077417 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:01:28.077556 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:01:28.077701 (+ 145us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:01:28.077701 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:01:28.077702 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:01:28.077750 (+ 48us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:01:28.077942 (+ 192us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 09:01:28.077943 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 09:01:28.077977 (+ 34us) client_negotiation.cc:769] Sending connection context
0619 09:01:28.078036 (+ 59us) client_negotiation.cc:240] Negotiation successful
0619 09:01:28.078127 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260619 09:01:28.078274 7240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:28.073127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34868 (local address 127.0.98.60:39539)
0619 09:01:28.073158 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:28.073160 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:28.073168 (+ 8us) server_negotiation.cc:407] Connection header received
0619 09:01:28.073216 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:28.073217 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:28.073223 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:28.073260 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:01:28.073451 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:28.073882 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:28.074470 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:28.074673 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:28.074728 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:28.076261 (+ 1533us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:01:28.076263 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:01:28.076263 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:01:28.076269 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:01:28.077349 (+ 1080us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:01:28.077609 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:01:28.077610 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:01:28.077610 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:01:28.077653 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:01:28.077800 (+ 147us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:01:28.077801 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:01:28.077801 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:01:28.077891 (+ 90us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:01:28.077941 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0619 09:01:28.078055 (+ 114us) server_negotiation.cc:299] Negotiation successful
0619 09:01:28.078171 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":62,"server-negotiator.queue_time_us":15}
I20260619 09:01:28.079653 7237 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f1435fb0f12c416196a176e829751c3b: Bootstrap starting.
I20260619 09:01:28.080475 7237 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f1435fb0f12c416196a176e829751c3b: Neither blocks nor log segments found. Creating new log.
I20260619 09:01:28.080721 7237 log.cc:824] T 00000000000000000000000000000000 P f1435fb0f12c416196a176e829751c3b: Log is configured to *not* fsync() on all Append() calls
I20260619 09:01:28.081429 7237 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f1435fb0f12c416196a176e829751c3b: No bootstrap required, opened a new log
I20260619 09:01:28.082799 7237 raft_consensus.cc:348] T 00000000000000000000000000000000 P f1435fb0f12c416196a176e829751c3b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aaa8435622414a159f995f651c058e08" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44403 } } peers { permanent_uuid: "13ea0fb4e8094bd8bda6763a0a07c2c5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34401 } } peers { permanent_uuid: "f1435fb0f12c416196a176e829751c3b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39539 } }
I20260619 09:01:28.082974 7237 raft_consensus.cc:374] T 00000000000000000000000000000000 P f1435fb0f12c416196a176e829751c3b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 09:01:28.083010 7237 raft_consensus.cc:729] T 00000000000000000000000000000000 P f1435fb0f12c416196a176e829751c3b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1435fb0f12c416196a176e829751c3b, State: Initialized, Role: FOLLOWER
I20260619 09:01:28.083132 7237 consensus_queue.cc:260] T 00000000000000000000000000000000 P f1435fb0f12c416196a176e829751c3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aaa8435622414a159f995f651c058e08" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44403 } } peers { permanent_uuid: "13ea0fb4e8094bd8bda6763a0a07c2c5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34401 } } peers { permanent_uuid: "f1435fb0f12c416196a176e829751c3b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39539 } }
I20260619 09:01:28.083384 7242 sys_catalog.cc:455] T 00000000000000000000000000000000 P f1435fb0f12c416196a176e829751c3b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aaa8435622414a159f995f651c058e08" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44403 } } peers { permanent_uuid: "13ea0fb4e8094bd8bda6763a0a07c2c5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34401 } } peers { permanent_uuid: "f1435fb0f12c416196a176e829751c3b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39539 } } }
I20260619 09:01:28.083431 7237 sys_catalog.cc:564] T 00000000000000000000000000000000 P f1435fb0f12c416196a176e829751c3b [sys.catalog]: configured and running, proceeding with master startup.
I20260619 09:01:28.083477 7242 sys_catalog.cc:458] T 00000000000000000000000000000000 P f1435fb0f12c416196a176e829751c3b [sys.catalog]: This master's current role is: FOLLOWER
W20260619 09:01:28.085028 7253 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f1435fb0f12c416196a176e829751c3b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 09:01:28.085070 7253 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 09:01:28.087142 7169 master_runner.cc:428] Detected that this master f1435fb0f12c416196a176e829751c3b is joining an existing cluster
I20260619 09:01:28.087191 7169 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.98.60:39539
I20260619 09:01:28.095758 7166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:28.088662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43262 (local address 127.0.98.61:34401)
0619 09:01:28.088715 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:28.088718 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:28.089210 (+ 492us) server_negotiation.cc:407] Connection header received
0619 09:01:28.089350 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:28.089351 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:28.089362 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:28.089393 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:01:28.089870 (+ 477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:28.090660 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:28.092091 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:28.092387 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:28.092422 (+ 35us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:28.093359 (+ 937us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:01:28.093360 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:01:28.093361 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:01:28.093367 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:01:28.094668 (+ 1301us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:01:28.095045 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:01:28.095046 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:01:28.095047 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:01:28.095101 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:01:28.095283 (+ 182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:01:28.095284 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:01:28.095284 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:01:28.095408 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:01:28.095447 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0619 09:01:28.095604 (+ 157us) server_negotiation.cc:299] Negotiation successful
0619 09:01:28.095669 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260619 09:01:28.095957 7260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:28.088549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34401 (local address 127.0.0.1:43262)
0619 09:01:28.089152 (+ 603us) negotiation.cc:107] Waiting for socket to connect
0619 09:01:28.089162 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 09:01:28.089264 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:01:28.089455 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:01:28.089456 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:01:28.089707 (+ 251us) client_negotiation.cc:189] Negotiated authn=SASL
0619 09:01:28.089783 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:28.089785 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:28.090780 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:28.090781 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:28.092005 (+ 1224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:28.092007 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:28.092482 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:28.092483 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:28.092554 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:28.092955 (+ 401us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 09:01:28.092960 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 09:01:28.093269 (+ 309us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 09:01:28.094752 (+ 1483us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:01:28.094753 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:01:28.094754 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:01:28.094982 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:01:28.095152 (+ 170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:01:28.095153 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:01:28.095153 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:01:28.095210 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:01:28.095484 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 09:01:28.095485 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 09:01:28.095517 (+ 32us) client_negotiation.cc:769] Sending connection context
0619 09:01:28.095566 (+ 49us) client_negotiation.cc:240] Negotiation successful
0619 09:01:28.095659 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":588,"mutex_wait_us":29,"thread_start_us":491,"threads_started":1}
I20260619 09:01:28.097074 6576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:28.088617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43060 (local address 127.0.98.62:44403)
0619 09:01:28.088665 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:28.088667 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:28.089192 (+ 525us) server_negotiation.cc:407] Connection header received
0619 09:01:28.089304 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:28.089305 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:28.089317 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:28.089349 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:01:28.089924 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:28.090668 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:28.092299 (+ 1631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:28.093209 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:28.093261 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:28.094441 (+ 1180us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:01:28.094443 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:01:28.094443 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:01:28.094451 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:01:28.096019 (+ 1568us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:01:28.096398 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:01:28.096399 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:01:28.096400 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:01:28.096460 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:01:28.096637 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:01:28.096638 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:01:28.096638 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:01:28.096742 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:01:28.096794 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0619 09:01:28.096919 (+ 125us) server_negotiation.cc:299] Negotiation successful
0619 09:01:28.096984 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260619 09:01:28.097087 7259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:28.088545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44403 (local address 127.0.0.1:43060)
0619 09:01:28.089118 (+ 573us) negotiation.cc:107] Waiting for socket to connect
0619 09:01:28.089130 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 09:01:28.089223 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:01:28.089426 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:01:28.089426 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:01:28.089729 (+ 303us) client_negotiation.cc:189] Negotiated authn=SASL
0619 09:01:28.089801 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:28.089804 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:28.090808 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:28.090809 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:28.092184 (+ 1375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:28.092186 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:28.093349 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:28.093350 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:28.093411 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:28.093953 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 09:01:28.093958 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 09:01:28.094332 (+ 374us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 09:01:28.096116 (+ 1784us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:01:28.096117 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:01:28.096118 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:01:28.096321 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:01:28.096530 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:01:28.096531 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:01:28.096531 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:01:28.096587 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:01:28.096793 (+ 206us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 09:01:28.096794 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 09:01:28.096832 (+ 38us) client_negotiation.cc:769] Sending connection context
0619 09:01:28.096893 (+ 61us) client_negotiation.cc:240] Negotiation successful
0619 09:01:28.096983 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":546,"thread_start_us":497,"threads_started":1}
Jun 19 09:01:28 dist-test-slave-mtdr krb5kdc[6472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859687, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260619 09:01:28.113979 7261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:28.089263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39539 (local address 127.0.0.1:34872)
0619 09:01:28.089365 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0619 09:01:28.089373 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0619 09:01:28.089435 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:01:28.089562 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:01:28.089563 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:01:28.089832 (+ 269us) client_negotiation.cc:189] Negotiated authn=SASL
0619 09:01:28.089880 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:28.089881 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:28.090723 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:28.090723 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:28.092005 (+ 1282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:28.092007 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:28.092467 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:28.092468 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:28.092541 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:28.111240 (+ 18699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 09:01:28.111245 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 09:01:28.111597 (+ 352us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 09:01:28.113036 (+ 1439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:01:28.113037 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:01:28.113038 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:01:28.113181 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:01:28.113334 (+ 153us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:01:28.113335 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:01:28.113335 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:01:28.113387 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:01:28.113588 (+ 201us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 09:01:28.113588 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 09:01:28.113626 (+ 38us) client_negotiation.cc:769] Sending connection context
0619 09:01:28.113708 (+ 82us) client_negotiation.cc:240] Negotiation successful
0619 09:01:28.113809 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":91,"mutex_wait_us":337,"thread_start_us":60,"threads_started":1}
I20260619 09:01:28.113986 7240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:28.089221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34872 (local address 127.0.98.60:39539)
0619 09:01:28.089252 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:28.089253 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:28.089409 (+ 156us) server_negotiation.cc:407] Connection header received
0619 09:01:28.089490 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:28.089491 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:28.089497 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:28.089519 (+ 22us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:01:28.089945 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:28.090632 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:28.092088 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:28.092355 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:28.092392 (+ 37us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:28.111677 (+ 19285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:01:28.111678 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:01:28.111678 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:01:28.111685 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:01:28.112963 (+ 1278us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:01:28.113235 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:01:28.113236 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:01:28.113236 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:01:28.113283 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:01:28.113435 (+ 152us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:01:28.113436 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:01:28.113436 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:01:28.113539 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:01:28.113590 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0619 09:01:28.113750 (+ 160us) server_negotiation.cc:299] Negotiation successful
0619 09:01:28.113860 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":36,"server-negotiator.queue_time_us":17}
I20260619 09:01:28.114723 7262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:28.098895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34878 (local address 127.0.98.60:39539)
0619 09:01:28.099058 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:28.099059 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:28.099067 (+ 8us) server_negotiation.cc:407] Connection header received
0619 09:01:28.099100 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:28.099100 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:28.099111 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:28.099174 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:01:28.099407 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:28.100155 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:28.100942 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:28.101247 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:28.101349 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:28.102983 (+ 1634us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:01:28.102984 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:01:28.102984 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:01:28.102992 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:01:28.113568 (+ 10576us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:01:28.113996 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:01:28.113997 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:01:28.113998 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:01:28.114064 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:01:28.114246 (+ 182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:01:28.114247 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:01:28.114247 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:01:28.114355 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:01:28.114405 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0619 09:01:28.114552 (+ 147us) server_negotiation.cc:299] Negotiation successful
0619 09:01:28.114625 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":77,"threads_started":1}
I20260619 09:01:28.114718 7075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:28.098717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39539 (local address 127.0.0.1:34878)
0619 09:01:28.098773 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0619 09:01:28.098784 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 09:01:28.098845 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:01:28.099201 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:01:28.099201 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:01:28.099314 (+ 113us) client_negotiation.cc:189] Negotiated authn=SASL
0619 09:01:28.099349 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:28.099351 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:28.100280 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:28.100281 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:28.100874 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:28.100876 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:28.101324 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:28.101325 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:28.101389 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:28.101735 (+ 346us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 09:01:28.101740 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 09:01:28.102911 (+ 1171us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 09:01:28.113699 (+ 10788us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:01:28.113700 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:01:28.113700 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:01:28.113917 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:01:28.114136 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:01:28.114137 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:01:28.114137 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:01:28.114195 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:01:28.114417 (+ 222us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 09:01:28.114418 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 09:01:28.114464 (+ 46us) client_negotiation.cc:769] Sending connection context
0619 09:01:28.114523 (+ 59us) client_negotiation.cc:240] Negotiation successful
0619 09:01:28.114625 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260619 09:01:28.115267 6517 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.98.60:39539: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "f1435fb0f12c416196a176e829751c3b"
member_type: NON_VOTER
last_known_addr {
host: "127.0.98.60"
port: 39539
}
attrs {
promote: true
}
}
dest_uuid: "aaa8435622414a159f995f651c058e08"
cas_config_opid_index: 5
I20260619 09:01:28.115495 6517 consensus_queue.cc:237] T 00000000000000000000000000000000 P aaa8435622414a159f995f651c058e08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "aaa8435622414a159f995f651c058e08" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44403 } } peers { permanent_uuid: "13ea0fb4e8094bd8bda6763a0a07c2c5" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 34401 } attrs { promote: true } } peers { permanent_uuid: "f1435fb0f12c416196a176e829751c3b" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39539 } attrs { promote: true } }
I20260619 09:01:28.115806 7077 raft_consensus.cc:2932] T 00000000000000000000000000000000 P aaa8435622414a159f995f651c058e08 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER f1435fb0f12c416196a176e829751c3b (127.0.98.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "aaa8435622414a159f995f651c058e08" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44403 } } peers { permanent_uuid: "13ea0fb4e8094bd8bda6763a0a07c2c5" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 34401 } attrs { promote: true } } peers { permanent_uuid: "f1435fb0f12c416196a176e829751c3b" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39539 } attrs { promote: true } } }
I20260619 09:01:28.116146 7211 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f1435fb0f12c416196a176e829751c3b [term 0 FOLLOWER]: Advancing to term 1
I20260619 09:01:28.116642 7077 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.98.60:39539
I20260619 09:01:28.116714 7077 sys_catalog.cc:455] T 00000000000000000000000000000000 P aaa8435622414a159f995f651c058e08 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "aaa8435622414a159f995f651c058e08" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "aaa8435622414a159f995f651c058e08" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44403 } } peers { permanent_uuid: "13ea0fb4e8094bd8bda6763a0a07c2c5" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 34401 } attrs { promote: true } } peers { permanent_uuid: "f1435fb0f12c416196a176e829751c3b" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39539 } attrs { promote: true } } }
I20260619 09:01:28.116786 7077 sys_catalog.cc:458] T 00000000000000000000000000000000 P aaa8435622414a159f995f651c058e08 [sys.catalog]: This master's current role is: LEADER
I20260619 09:01:28.117022 7211 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f1435fb0f12c416196a176e829751c3b [term 1 FOLLOWER]: Refusing update from remote peer aaa8435622414a159f995f651c058e08: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260619 09:01:28.117061 7169 master.cc:537] Master@127.0.98.60:39539 shutting down...
I20260619 09:01:28.118019 7076 consensus_queue.cc:1035] T 00000000000000000000000000000000 P aaa8435622414a159f995f651c058e08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1435fb0f12c416196a176e829751c3b" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39539 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
W20260619 09:01:28.118793 6504 consensus_peers.cc:487] T 00000000000000000000000000000000 P aaa8435622414a159f995f651c058e08 -> Peer f1435fb0f12c416196a176e829751c3b (127.0.98.60:39539): Couldn't send request to peer f1435fb0f12c416196a176e829751c3b. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260619 09:01:28.120723 7169 raft_consensus.cc:2229] T 00000000000000000000000000000000 P f1435fb0f12c416196a176e829751c3b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 09:01:28.120827 7169 raft_consensus.cc:2258] T 00000000000000000000000000000000 P f1435fb0f12c416196a176e829751c3b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 09:01:28.120883 7169 tablet_replica.cc:330] T 00000000000000000000000000000000 P f1435fb0f12c416196a176e829751c3b: stopping tablet replica
W20260619 09:01:28.122074 6470 connection.cc:535] client connection to 127.0.98.60:39539 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260619 09:01:28.122299 7166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:28.116863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43264 (local address 127.0.98.61:34401)
0619 09:01:28.116902 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:28.116903 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:28.116913 (+ 10us) server_negotiation.cc:407] Connection header received
0619 09:01:28.116936 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:28.116937 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:28.116959 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:28.117014 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 09:01:28.117656 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:28.118349 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:28.121176 (+ 2827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:28.121719 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:28.121785 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:28.121872 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0619 09:01:28.122197 (+ 325us) server_negotiation.cc:299] Negotiation successful
0619 09:01:28.122235 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260619 09:01:28.122125 7075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:28.115886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34401 (local address 127.0.0.1:43264)
0619 09:01:28.116035 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0619 09:01:28.116044 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 09:01:28.116081 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:01:28.117072 (+ 991us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:01:28.117073 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:01:28.117075 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 09:01:28.117116 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:28.117117 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:28.118507 (+ 1390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:28.118508 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:28.120970 (+ 2462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:28.120974 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:28.121904 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:28.121905 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:28.121958 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:28.121975 (+ 17us) client_negotiation.cc:769] Sending connection context
0619 09:01:28.122021 (+ 46us) client_negotiation.cc:240] Negotiation successful
0619 09:01:28.122062 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134}
I20260619 09:01:28.123756 7125 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 13ea0fb4e8094bd8bda6763a0a07c2c5 [term 1 LEARNER]: Refusing update from remote peer aaa8435622414a159f995f651c058e08: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260619 09:01:28.123986 7076 consensus_queue.cc:1035] T 00000000000000000000000000000000 P aaa8435622414a159f995f651c058e08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "13ea0fb4e8094bd8bda6763a0a07c2c5" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 34401 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260619 09:01:28.124521 7125 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 13ea0fb4e8094bd8bda6763a0a07c2c5 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER f1435fb0f12c416196a176e829751c3b (127.0.98.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "aaa8435622414a159f995f651c058e08" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44403 } } peers { permanent_uuid: "13ea0fb4e8094bd8bda6763a0a07c2c5" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 34401 } attrs { promote: true } } peers { permanent_uuid: "f1435fb0f12c416196a176e829751c3b" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39539 } attrs { promote: true } } }
I20260619 09:01:28.125597 7264 sys_catalog.cc:455] T 00000000000000000000000000000000 P 13ea0fb4e8094bd8bda6763a0a07c2c5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "aaa8435622414a159f995f651c058e08" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "aaa8435622414a159f995f651c058e08" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44403 } } peers { permanent_uuid: "13ea0fb4e8094bd8bda6763a0a07c2c5" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 34401 } attrs { promote: true } } peers { permanent_uuid: "f1435fb0f12c416196a176e829751c3b" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39539 } attrs { promote: true } } }
I20260619 09:01:28.125685 7264 sys_catalog.cc:458] T 00000000000000000000000000000000 P 13ea0fb4e8094bd8bda6763a0a07c2c5 [sys.catalog]: This master's current role is: LEARNER
I20260619 09:01:28.125949 7154 sys_catalog.cc:455] T 00000000000000000000000000000000 P 13ea0fb4e8094bd8bda6763a0a07c2c5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader aaa8435622414a159f995f651c058e08. Latest consensus state: current_term: 1 leader_uuid: "aaa8435622414a159f995f651c058e08" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "aaa8435622414a159f995f651c058e08" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44403 } } peers { permanent_uuid: "13ea0fb4e8094bd8bda6763a0a07c2c5" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 34401 } attrs { promote: true } } peers { permanent_uuid: "f1435fb0f12c416196a176e829751c3b" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39539 } attrs { promote: true } } }
I20260619 09:01:28.126031 7154 sys_catalog.cc:458] T 00000000000000000000000000000000 P 13ea0fb4e8094bd8bda6763a0a07c2c5 [sys.catalog]: This master's current role is: LEARNER
I20260619 09:01:28.126147 7076 raft_consensus.cc:1050] T 00000000000000000000000000000000 P aaa8435622414a159f995f651c058e08: attempting to promote NON_VOTER 13ea0fb4e8094bd8bda6763a0a07c2c5 to VOTER
I20260619 09:01:28.126350 7076 consensus_queue.cc:237] T 00000000000000000000000000000000 P aaa8435622414a159f995f651c058e08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "aaa8435622414a159f995f651c058e08" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44403 } } peers { permanent_uuid: "13ea0fb4e8094bd8bda6763a0a07c2c5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34401 } attrs { promote: false } } peers { permanent_uuid: "f1435fb0f12c416196a176e829751c3b" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39539 } attrs { promote: true } }
I20260619 09:01:28.126978 7125 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 13ea0fb4e8094bd8bda6763a0a07c2c5 [term 1 LEARNER]: Refusing update from remote peer aaa8435622414a159f995f651c058e08: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260619 09:01:28.127215 7265 consensus_queue.cc:1035] T 00000000000000000000000000000000 P aaa8435622414a159f995f651c058e08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "13ea0fb4e8094bd8bda6763a0a07c2c5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34401 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260619 09:01:28.127746 7075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:28.126794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39539 (local address 127.0.0.1:34888)
0619 09:01:28.127646 (+ 852us) negotiation.cc:107] Waiting for socket to connect
0619 09:01:28.127702 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":833}
I20260619 09:01:28.128177 7075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:28.128031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39539 (local address 127.0.0.1:34902)
0619 09:01:28.128127 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0619 09:01:28.128153 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
W20260619 09:01:28.128234 6504 consensus_peers.cc:487] T 00000000000000000000000000000000 P aaa8435622414a159f995f651c058e08 -> Peer f1435fb0f12c416196a176e829751c3b (127.0.98.60:39539): Couldn't send request to peer f1435fb0f12c416196a176e829751c3b. Status: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39539: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260619 09:01:28.128466 7077 raft_consensus.cc:2932] T 00000000000000000000000000000000 P aaa8435622414a159f995f651c058e08 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 13ea0fb4e8094bd8bda6763a0a07c2c5 (127.0.98.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "aaa8435622414a159f995f651c058e08" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44403 } } peers { permanent_uuid: "13ea0fb4e8094bd8bda6763a0a07c2c5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34401 } attrs { promote: false } } peers { permanent_uuid: "f1435fb0f12c416196a176e829751c3b" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39539 } attrs { promote: true } } }
I20260619 09:01:28.129127 7077 sys_catalog.cc:455] T 00000000000000000000000000000000 P aaa8435622414a159f995f651c058e08 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "aaa8435622414a159f995f651c058e08" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "aaa8435622414a159f995f651c058e08" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44403 } } peers { permanent_uuid: "13ea0fb4e8094bd8bda6763a0a07c2c5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34401 } attrs { promote: false } } peers { permanent_uuid: "f1435fb0f12c416196a176e829751c3b" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39539 } attrs { promote: true } } }
I20260619 09:01:28.129201 7077 sys_catalog.cc:458] T 00000000000000000000000000000000 P aaa8435622414a159f995f651c058e08 [sys.catalog]: This master's current role is: LEADER
I20260619 09:01:28.129150 7265 sys_catalog.cc:455] T 00000000000000000000000000000000 P aaa8435622414a159f995f651c058e08 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "aaa8435622414a159f995f651c058e08" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "aaa8435622414a159f995f651c058e08" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44403 } } peers { permanent_uuid: "13ea0fb4e8094bd8bda6763a0a07c2c5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34401 } attrs { promote: false } } peers { permanent_uuid: "f1435fb0f12c416196a176e829751c3b" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39539 } attrs { promote: true } } }
I20260619 09:01:28.129253 7265 sys_catalog.cc:458] T 00000000000000000000000000000000 P aaa8435622414a159f995f651c058e08 [sys.catalog]: This master's current role is: LEADER
I20260619 09:01:28.129436 7125 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 13ea0fb4e8094bd8bda6763a0a07c2c5 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 13ea0fb4e8094bd8bda6763a0a07c2c5 (127.0.98.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "aaa8435622414a159f995f651c058e08" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44403 } } peers { permanent_uuid: "13ea0fb4e8094bd8bda6763a0a07c2c5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34401 } attrs { promote: false } } peers { permanent_uuid: "f1435fb0f12c416196a176e829751c3b" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39539 } attrs { promote: true } } }
I20260619 09:01:28.130251 7154 sys_catalog.cc:455] T 00000000000000000000000000000000 P 13ea0fb4e8094bd8bda6763a0a07c2c5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "aaa8435622414a159f995f651c058e08" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "aaa8435622414a159f995f651c058e08" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44403 } } peers { permanent_uuid: "13ea0fb4e8094bd8bda6763a0a07c2c5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34401 } attrs { promote: false } } peers { permanent_uuid: "f1435fb0f12c416196a176e829751c3b" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39539 } attrs { promote: true } } }
I20260619 09:01:28.130347 7154 sys_catalog.cc:458] T 00000000000000000000000000000000 P 13ea0fb4e8094bd8bda6763a0a07c2c5 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 09:01:28.133599 7169 master.cc:552] Master@127.0.98.60:39539 shutdown complete.
I20260619 09:01:28.133638 7169 master_runner.cc:305] Clearing existing system tablet
I20260619 09:01:28.133742 7169 fs_manager.cc:359] Metadata directory not provided
I20260619 09:01:28.133781 7169 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 09:01:28.134336 7169 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.001s
I20260619 09:01:28.134752 7267 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 09:01:28.134939 7169 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859658134510-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 09:01:28.134991 7169 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260619 09:01:28.135013 7169 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859658134510-392-0/minicluster-data/master-2/data,/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859658134510-392-0/minicluster-data/master-2/wal
uuid: "f1435fb0f12c416196a176e829751c3b"
format_stamp: "Formatted at 2026-06-19 09:01:28 on dist-test-slave-mtdr"
server_key: "c91f639a49b8dea10b67aa055d0638b7"
server_key_iv: "a309b464eb8d861c61a731fee2058a98"
server_key_version: "encryptionkey@0"
I20260619 09:01:28.135569 7169 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P f1435fb0f12c416196a176e829751c3b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 09:01:28.136659 7169 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P f1435fb0f12c416196a176e829751c3b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260619 09:01:28.136715 7169 log.cc:1196] T 00000000000000000000000000000000 P f1435fb0f12c416196a176e829751c3b: Deleting WAL directory at /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859658134510-392-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260619 09:01:28.136866 7169 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P f1435fb0f12c416196a176e829751c3b: Deleting consensus metadata
I20260619 09:01:28.136978 7169 master_runner.cc:315] Copying system tablet from 127.0.98.62:44403
I20260619 09:01:28.137495 7169 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P f1435fb0f12c416196a176e829751c3b: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.62:44403
I20260619 09:01:28.143561 6576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:28.137814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43064 (local address 127.0.98.62:44403)
0619 09:01:28.137856 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:28.137858 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:28.137916 (+ 58us) server_negotiation.cc:407] Connection header received
0619 09:01:28.137990 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:28.137991 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:28.137999 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:28.138042 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:01:28.138413 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:28.138958 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:28.139973 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:28.140155 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:28.140214 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:28.141132 (+ 918us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:01:28.141133 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:01:28.141133 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:01:28.141139 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:01:28.142491 (+ 1352us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:01:28.142933 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:01:28.142934 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:01:28.142935 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:01:28.142983 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:01:28.143156 (+ 173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:01:28.143156 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:01:28.143157 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:01:28.143251 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:01:28.143294 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0619 09:01:28.143441 (+ 147us) server_negotiation.cc:299] Negotiation successful
0619 09:01:28.143499 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260619 09:01:28.143630 7273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:28.137736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44403 (local address 127.0.0.1:43064)
0619 09:01:28.137855 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0619 09:01:28.137866 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 09:01:28.137933 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:01:28.138055 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:01:28.138056 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:01:28.138249 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0619 09:01:28.138343 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:28.138346 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:28.139022 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:28.139023 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:28.139878 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:28.139881 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:28.140222 (+ 341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:28.140223 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:28.140281 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:28.140684 (+ 403us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 09:01:28.140689 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 09:01:28.141058 (+ 369us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 09:01:28.142566 (+ 1508us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:01:28.142567 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:01:28.142567 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:01:28.142776 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:01:28.143039 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:01:28.143039 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:01:28.143040 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:01:28.143104 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:01:28.143299 (+ 195us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 09:01:28.143299 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 09:01:28.143352 (+ 53us) client_negotiation.cc:769] Sending connection context
0619 09:01:28.143423 (+ 71us) client_negotiation.cc:240] Negotiation successful
0619 09:01:28.143520 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":62,"threads_started":1}
I20260619 09:01:28.143922 6537 tablet_copy_service.cc:140] P aaa8435622414a159f995f651c058e08: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer f1435fb0f12c416196a176e829751c3b ({username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:43064)
I20260619 09:01:28.143986 6537 tablet_copy_service.cc:161] P aaa8435622414a159f995f651c058e08: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer f1435fb0f12c416196a176e829751c3b at {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:43064: session id = f1435fb0f12c416196a176e829751c3b-00000000000000000000000000000000
I20260619 09:01:28.144577 6537 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P aaa8435622414a159f995f651c058e08: Tablet Copy: opened 0 blocks and 1 log segments
I20260619 09:01:28.145022 7169 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260619 09:01:28.145933 7169 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P f1435fb0f12c416196a176e829751c3b: tablet copy: Starting download of 0 data blocks...
I20260619 09:01:28.146061 7169 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P f1435fb0f12c416196a176e829751c3b: tablet copy: Starting download of 1 WAL segments...
I20260619 09:01:28.146838 7169 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P f1435fb0f12c416196a176e829751c3b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260619 09:01:28.148005 6537 tablet_copy_service.cc:342] P aaa8435622414a159f995f651c058e08: Request end of tablet copy session f1435fb0f12c416196a176e829751c3b-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:43064
I20260619 09:01:28.148051 6537 tablet_copy_service.cc:433] P aaa8435622414a159f995f651c058e08: ending tablet copy session f1435fb0f12c416196a176e829751c3b-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer f1435fb0f12c416196a176e829751c3b
I20260619 09:01:28.150418 7169 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 09:01:28.151491 7275 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:01:28.151538 7277 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:01:28.151456 7274 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:01:28.155009 7169 server_base.cc:1034] running on GCE node
I20260619 09:01:28.155113 7169 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 09:01:28.155241 7169 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 09:01:28.156371 7169 hybrid_clock.cc:648] HybridClock initialized: now 1781859688156356 us; error 31 us; skew 500 ppm
Jun 19 09:01:28 dist-test-slave-mtdr krb5kdc[6472](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859688, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 09:01:28.157649 7169 init.cc:377] Logged in from keytab as oryx/127.0.98.60@KRBTEST.COM (short username oryx)
I20260619 09:01:28.157900 7169 webserver.cc:460] Webserver started at http://127.0.98.60:35305/ using document root <none> and password file <none>
I20260619 09:01:28.158002 7169 fs_manager.cc:359] Metadata directory not provided
I20260619 09:01:28.158036 7169 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 09:01:28.158565 7169 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.000s
I20260619 09:01:28.159060 7284 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 09:01:28.159209 7169 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260619 09:01:28.159257 7169 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859658134510-392-0/minicluster-data/master-2/data,/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859658134510-392-0/minicluster-data/master-2/wal
uuid: "f1435fb0f12c416196a176e829751c3b"
format_stamp: "Formatted at 2026-06-19 09:01:28 on dist-test-slave-mtdr"
server_key: "c91f639a49b8dea10b67aa055d0638b7"
server_key_iv: "a309b464eb8d861c61a731fee2058a98"
server_key_version: "encryptionkey@0"
I20260619 09:01:28.159314 7169 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859658134510-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859658134510-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859658134510-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 09:01:28.174284 7169 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 09:01:28.175761 7169 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 09:01:28.178037 7169 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:39539
I20260619 09:01:28.178184 7336 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:39539 every 8 connection(s)
I20260619 09:01:28.178367 7169 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859658134510-392-0/minicluster-data/master-2/data/info.pb
I20260619 09:01:28.179247 7337 sys_catalog.cc:263] Verifying existing consensus state
I20260619 09:01:28.179816 7337 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f1435fb0f12c416196a176e829751c3b: Bootstrap starting.
I20260619 09:01:28.182709 7337 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f1435fb0f12c416196a176e829751c3b: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 09:01:28.183056 7337 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f1435fb0f12c416196a176e829751c3b: Bootstrap complete.
I20260619 09:01:28.183261 7337 raft_consensus.cc:348] T 00000000000000000000000000000000 P f1435fb0f12c416196a176e829751c3b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "aaa8435622414a159f995f651c058e08" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44403 } } peers { permanent_uuid: "13ea0fb4e8094bd8bda6763a0a07c2c5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34401 } attrs { promote: false } } peers { permanent_uuid: "f1435fb0f12c416196a176e829751c3b" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39539 } attrs { promote: true } }
I20260619 09:01:28.183362 7337 raft_consensus.cc:729] T 00000000000000000000000000000000 P f1435fb0f12c416196a176e829751c3b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f1435fb0f12c416196a176e829751c3b, State: Initialized, Role: LEARNER
I20260619 09:01:28.183409 7337 consensus_queue.cc:260] T 00000000000000000000000000000000 P f1435fb0f12c416196a176e829751c3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "aaa8435622414a159f995f651c058e08" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44403 } } peers { permanent_uuid: "13ea0fb4e8094bd8bda6763a0a07c2c5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34401 } attrs { promote: false } } peers { permanent_uuid: "f1435fb0f12c416196a176e829751c3b" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39539 } attrs { promote: true } }
I20260619 09:01:28.183581 7337 sys_catalog.cc:564] T 00000000000000000000000000000000 P f1435fb0f12c416196a176e829751c3b [sys.catalog]: configured and running, proceeding with master startup.
I20260619 09:01:28.183588 7340 sys_catalog.cc:455] T 00000000000000000000000000000000 P f1435fb0f12c416196a176e829751c3b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "aaa8435622414a159f995f651c058e08" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44403 } } peers { permanent_uuid: "13ea0fb4e8094bd8bda6763a0a07c2c5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34401 } attrs { promote: false } } peers { permanent_uuid: "f1435fb0f12c416196a176e829751c3b" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39539 } attrs { promote: true } } }
I20260619 09:01:28.183805 7340 sys_catalog.cc:458] T 00000000000000000000000000000000 P f1435fb0f12c416196a176e829751c3b [sys.catalog]: This master's current role is: LEARNER
I20260619 09:01:28.185022 7351 catalog_manager.cc:1259] Loaded cluster ID: aac0526a83f348348927f89c7ec729d5
I20260619 09:01:28.185070 7351 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f1435fb0f12c416196a176e829751c3b: loading cluster ID for follower catalog manager: success
I20260619 09:01:28.186179 7351 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f1435fb0f12c416196a176e829751c3b: acquiring CA information for follower catalog manager: success
I20260619 09:01:28.186429 7351 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f1435fb0f12c416196a176e829751c3b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 09:01:28.229339 7352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:28.223112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34928 (local address 127.0.98.60:39539)
0619 09:01:28.223225 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:28.223227 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:28.223236 (+ 9us) server_negotiation.cc:407] Connection header received
0619 09:01:28.223260 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:28.223261 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:28.223269 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:28.223331 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:01:28.223999 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:28.224683 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:28.225605 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:28.225816 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:28.225872 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:28.227065 (+ 1193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:01:28.227067 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:01:28.227067 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:01:28.227074 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:01:28.228374 (+ 1300us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:01:28.228631 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:01:28.228632 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:01:28.228633 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:01:28.228687 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:01:28.228886 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:01:28.228886 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:01:28.228887 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:01:28.229002 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:01:28.229070 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0619 09:01:28.229211 (+ 141us) server_negotiation.cc:299] Negotiation successful
0619 09:01:28.229267 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94,"thread_start_us":57,"threads_started":1}
I20260619 09:01:28.229861 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ez5NB/build/release/bin/kudu with pid 6579
I20260619 09:01:28.234102 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ez5NB/build/release/bin/kudu
/tmp/dist-test-task4ez5NB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859658134510-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859658134510-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859658134510-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859658134510-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:40909
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:44403
--webserver_port=37141
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859658134510-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42929
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:44403,127.0.98.61:34401
--tserver_master_addrs=127.0.98.62:44403,127.0.98.61:34401,127.0.98.60:39539 with env {KRB5CCNAME=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859658134510-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859658134510-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859658134510-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 09:01:28.312572 7353 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 09:01:28.312806 7353 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 09:01:28.312829 7353 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 09:01:28.313943 7353 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 09:01:28.313992 7353 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 09:01:28.314016 7353 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 09:01:28.315472 7353 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42929
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859658134510-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859658134510-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859658134510-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:40909
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859658134510-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=37141
--tserver_master_addrs=127.0.98.62:44403,127.0.98.61:34401,127.0.98.60:39539
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7353
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859658134510-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 09:01:28.315691 7353 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 09:01:28.315903 7353 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 09:01:28.320562 7353 server_base.cc:1034] running on GCE node
W20260619 09:01:28.320595 7361 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:01:28.320537 7359 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:01:28.320547 7358 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:01:28.320920 7353 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 09:01:28.321146 7353 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 09:01:28.322283 7353 hybrid_clock.cc:648] HybridClock initialized: now 1781859688322269 us; error 31 us; skew 500 ppm
Jun 19 09:01:28 dist-test-slave-mtdr krb5kdc[6472](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859688, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 09:01:28.324657 7353 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260619 09:01:28.325043 7353 webserver.cc:460] Webserver started at http://127.0.98.1:37141/ using document root <none> and password file <none>
I20260619 09:01:28.325239 7353 fs_manager.cc:359] Metadata directory not provided
I20260619 09:01:28.325292 7353 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 09:01:28.326558 7353 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 09:01:28.327085 7368 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 09:01:28.327224 7353 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 09:01:28.327302 7353 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859658134510-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859658134510-392-0/minicluster-data/ts-0/wal
uuid: "551feb9174794395a270058ce5ed45c5"
format_stamp: "Formatted at 2026-06-19 09:01:27 on dist-test-slave-mtdr"
server_key: "900dbdbbd43114037e85d9c7fa4444d0"
server_key_iv: "8baf4b3c5db85ad229d3e9464408e366"
server_key_version: "encryptionkey@0"
I20260619 09:01:28.327618 7353 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859658134510-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859658134510-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859658134510-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 09:01:28.337762 7353 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 09:01:28.339489 7353 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 09:01:28.339600 7353 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 09:01:28.339780 7353 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 09:01:28.340050 7353 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 09:01:28.340082 7353 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 09:01:28.340108 7353 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 09:01:28.340133 7353 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 09:01:28.345690 7353 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40909
I20260619 09:01:28.345750 7481 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40909 every 8 connection(s)
I20260619 09:01:28.346042 7353 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859658134510-392-0/minicluster-data/ts-0/data/info.pb
I20260619 09:01:28.348295 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ez5NB/build/release/bin/kudu as pid 7353
I20260619 09:01:28.348443 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ez5NB/build/release/bin/kudu with pid 6713
Jun 19 09:01:28 dist-test-slave-mtdr krb5kdc[6472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859688, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
Jun 19 09:01:28 dist-test-slave-mtdr krb5kdc[6472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859688, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Jun 19 09:01:28 dist-test-slave-mtdr krb5kdc[6472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859688, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260619 09:01:28.355484 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ez5NB/build/release/bin/kudu
/tmp/dist-test-task4ez5NB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859658134510-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859658134510-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859658134510-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859658134510-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:37741
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:44403
--webserver_port=34281
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859658134510-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42929
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:44403,127.0.98.61:34401
--tserver_master_addrs=127.0.98.62:44403,127.0.98.61:34401,127.0.98.60:39539 with env {KRB5CCNAME=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859658134510-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859658134510-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859658134510-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 09:01:28.356760 7352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:28.346668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50937 (local address 127.0.98.60:39539)
0619 09:01:28.346717 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:28.346719 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:28.346960 (+ 241us) server_negotiation.cc:407] Connection header received
0619 09:01:28.347539 (+ 579us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:28.347541 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:28.347556 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:28.347605 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:01:28.348404 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:28.349113 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:28.350664 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:28.350993 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:28.351035 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:28.353732 (+ 2697us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:01:28.353734 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:01:28.353734 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:01:28.353742 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:01:28.355290 (+ 1548us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:01:28.355722 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:01:28.355723 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:01:28.355724 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:01:28.355778 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:01:28.356053 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:01:28.356054 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:01:28.356054 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:01:28.356182 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:01:28.356222 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0619 09:01:28.356594 (+ 372us) server_negotiation.cc:299] Negotiation successful
0619 09:01:28.356672 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260619 09:01:28.356940 7486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:28.346638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39539 (local address 127.0.98.1:50937)
0619 09:01:28.346845 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0619 09:01:28.346885 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 09:01:28.347441 (+ 556us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:01:28.347677 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:01:28.347681 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:01:28.348017 (+ 336us) client_negotiation.cc:189] Negotiated authn=SASL
0619 09:01:28.348293 (+ 276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:28.348299 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:28.349282 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:28.349283 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:28.350570 (+ 1287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:28.350573 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:28.351116 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:28.351117 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:28.351248 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:28.352042 (+ 794us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 09:01:28.352047 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 09:01:28.353638 (+ 1591us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 09:01:28.355370 (+ 1732us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:01:28.355377 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:01:28.355379 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:01:28.355647 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:01:28.355849 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:01:28.355849 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:01:28.355850 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:01:28.355968 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:01:28.356264 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 09:01:28.356269 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 09:01:28.356421 (+ 152us) client_negotiation.cc:769] Sending connection context
0619 09:01:28.356573 (+ 152us) client_negotiation.cc:240] Negotiation successful
0619 09:01:28.356723 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"mutex_wait_us":36,"thread_start_us":69,"threads_started":1}
I20260619 09:01:28.357479 7487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:28.346682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44403 (local address 127.0.98.1:35165)
0619 09:01:28.346878 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0619 09:01:28.346888 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 09:01:28.347445 (+ 557us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:01:28.348042 (+ 597us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:01:28.348043 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:01:28.348166 (+ 123us) client_negotiation.cc:189] Negotiated authn=SASL
0619 09:01:28.348293 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:28.348299 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:28.349291 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:28.349291 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:28.350571 (+ 1280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:28.350573 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:28.351110 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:28.351111 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:28.351248 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:28.351862 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 09:01:28.351869 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0619 09:01:28.354216 (+ 2347us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 09:01:28.356374 (+ 2158us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:01:28.356375 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:01:28.356375 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:01:28.356625 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:01:28.356835 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:01:28.356835 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:01:28.356836 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:01:28.356889 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:01:28.357154 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 09:01:28.357154 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 09:01:28.357196 (+ 42us) client_negotiation.cc:769] Sending connection context
0619 09:01:28.357278 (+ 82us) client_negotiation.cc:240] Negotiation successful
0619 09:01:28.357372 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":181,"mutex_wait_us":44,"thread_start_us":73,"threads_started":1}
I20260619 09:01:28.357482 6576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:28.346739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35165 (local address 127.0.98.62:44403)
0619 09:01:28.346800 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:28.346802 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:28.346954 (+ 152us) server_negotiation.cc:407] Connection header received
0619 09:01:28.347548 (+ 594us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:28.347549 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:28.347560 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:28.347588 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:01:28.348448 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:28.349168 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:28.350670 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:28.350985 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:28.351024 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:28.354308 (+ 3284us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:01:28.354310 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:01:28.354310 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:01:28.354317 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:01:28.356286 (+ 1969us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:01:28.356709 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:01:28.356710 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:01:28.356711 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:01:28.356768 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:01:28.356951 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:01:28.356951 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:01:28.356952 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:01:28.357067 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:01:28.357103 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0619 09:01:28.357320 (+ 217us) server_negotiation.cc:299] Negotiation successful
0619 09:01:28.357399 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260619 09:01:28.357632 7482 heartbeater.cc:347] Connected to a master server at 127.0.98.60:39539
I20260619 09:01:28.357699 7482 heartbeater.cc:464] Registering TS with master...
I20260619 09:01:28.357827 7482 heartbeater.cc:511] Master 127.0.98.60:39539 requested a full tablet report, sending...
I20260619 09:01:28.358176 7483 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44403
I20260619 09:01:28.358218 7483 heartbeater.cc:464] Registering TS with master...
I20260619 09:01:28.358289 7483 heartbeater.cc:511] Master 127.0.98.62:44403 requested a full tablet report, sending...
I20260619 09:01:28.358342 7301 ts_manager.cc:194] Registered new tserver with Master: 551feb9174794395a270058ce5ed45c5 (127.0.98.1:40909)
I20260619 09:01:28.358536 6517 ts_manager.cc:194] Re-registered known tserver with Master: 551feb9174794395a270058ce5ed45c5 (127.0.98.1:40909)
I20260619 09:01:28.358933 6517 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:35165
I20260619 09:01:28.360065 7166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:28.346991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33797 (local address 127.0.98.61:34401)
0619 09:01:28.347037 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:28.347038 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:28.347166 (+ 128us) server_negotiation.cc:407] Connection header received
0619 09:01:28.347604 (+ 438us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:28.347605 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:28.347615 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:28.347670 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:01:28.348394 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:28.349163 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:28.350669 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:28.351001 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:28.351034 (+ 33us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:28.354878 (+ 3844us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:01:28.354879 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:01:28.354879 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:01:28.354887 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:01:28.356307 (+ 1420us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:01:28.359325 (+ 3018us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:01:28.359327 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:01:28.359327 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:01:28.359386 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:01:28.359594 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:01:28.359595 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:01:28.359596 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:01:28.359712 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:01:28.359766 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0619 09:01:28.359904 (+ 138us) server_negotiation.cc:299] Negotiation successful
0619 09:01:28.359971 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260619 09:01:28.360085 7488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:28.346920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34401 (local address 127.0.98.1:33797)
0619 09:01:28.347107 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0619 09:01:28.347115 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0619 09:01:28.347453 (+ 338us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:01:28.347694 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:01:28.347695 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:01:28.348028 (+ 333us) client_negotiation.cc:189] Negotiated authn=SASL
0619 09:01:28.348293 (+ 265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:28.348298 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:28.349288 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:28.349288 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:28.350591 (+ 1303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:28.350592 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:28.351112 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:28.351113 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:28.351248 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:28.351925 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 09:01:28.351929 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 09:01:28.354758 (+ 2829us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 09:01:28.356374 (+ 1616us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:01:28.356375 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:01:28.356375 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:01:28.359245 (+ 2870us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:01:28.359452 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:01:28.359453 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:01:28.359453 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:01:28.359518 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:01:28.359769 (+ 251us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 09:01:28.359770 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 09:01:28.359807 (+ 37us) client_negotiation.cc:769] Sending connection context
0619 09:01:28.359866 (+ 59us) client_negotiation.cc:240] Negotiation successful
0619 09:01:28.359972 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173,"mutex_wait_us":52,"thread_start_us":66,"threads_started":1}
I20260619 09:01:28.360358 7484 heartbeater.cc:347] Connected to a master server at 127.0.98.61:34401
I20260619 09:01:28.360404 7484 heartbeater.cc:464] Registering TS with master...
I20260619 09:01:28.360463 7484 heartbeater.cc:511] Master 127.0.98.61:34401 requested a full tablet report, sending...
I20260619 09:01:28.360749 7115 ts_manager.cc:194] Registered new tserver with Master: 551feb9174794395a270058ce5ed45c5 (127.0.98.1:40909)
W20260619 09:01:28.435202 7489 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 09:01:28.435395 7489 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 09:01:28.435416 7489 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 09:01:28.436527 7489 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 09:01:28.436565 7489 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 09:01:28.436590 7489 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 09:01:28.438020 7489 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42929
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859658134510-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859658134510-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859658134510-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:37741
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859658134510-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=34281
--tserver_master_addrs=127.0.98.62:44403,127.0.98.61:34401,127.0.98.60:39539
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7489
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859658134510-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 09:01:28.438239 7489 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 09:01:28.438454 7489 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 09:01:28.443089 7497 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:01:28.443212 7489 server_base.cc:1034] running on GCE node
W20260619 09:01:28.443099 7495 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:01:28.443091 7494 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:01:28.443490 7489 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 09:01:28.443699 7489 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 09:01:28.444829 7489 hybrid_clock.cc:648] HybridClock initialized: now 1781859688444813 us; error 26 us; skew 500 ppm
Jun 19 09:01:28 dist-test-slave-mtdr krb5kdc[6472](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859688, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 09:01:28.447113 7489 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260619 09:01:28.447513 7489 webserver.cc:460] Webserver started at http://127.0.98.2:34281/ using document root <none> and password file <none>
I20260619 09:01:28.447718 7489 fs_manager.cc:359] Metadata directory not provided
I20260619 09:01:28.447779 7489 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 09:01:28.448985 7489 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 09:01:28.449486 7504 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 09:01:28.449661 7489 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 09:01:28.449756 7489 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859658134510-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859658134510-392-0/minicluster-data/ts-1/wal
uuid: "00cca0af6ec34749baea4d021c20755e"
format_stamp: "Formatted at 2026-06-19 09:01:27 on dist-test-slave-mtdr"
server_key: "247defef2043a8dcc0ac213830a070ef"
server_key_iv: "7e262f62ec1186890ade9b148212dbe3"
server_key_version: "encryptionkey@0"
I20260619 09:01:28.450086 7489 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859658134510-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859658134510-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859658134510-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 09:01:28.474472 7489 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 09:01:28.476331 7489 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 09:01:28.476476 7489 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 09:01:28.476755 7489 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 09:01:28.477072 7489 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 09:01:28.477118 7489 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 09:01:28.477144 7489 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 09:01:28.477166 7489 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 09:01:28.482679 7489 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37741
I20260619 09:01:28.482750 7617 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37741 every 8 connection(s)
I20260619 09:01:28.483057 7489 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859658134510-392-0/minicluster-data/ts-1/data/info.pb
Jun 19 09:01:28 dist-test-slave-mtdr krb5kdc[6472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859688, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
Jun 19 09:01:28 dist-test-slave-mtdr krb5kdc[6472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859688, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260619 09:01:28.490847 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ez5NB/build/release/bin/kudu as pid 7489
I20260619 09:01:28.490989 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ez5NB/build/release/bin/kudu with pid 6847
Jun 19 09:01:28 dist-test-slave-mtdr krb5kdc[6472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859688, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260619 09:01:28.494153 6576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:28.484040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36601 (local address 127.0.98.62:44403)
0619 09:01:28.484091 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:28.484093 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:28.484104 (+ 11us) server_negotiation.cc:407] Connection header received
0619 09:01:28.485109 (+ 1005us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:28.485110 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:28.485123 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:28.485184 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:01:28.486855 (+ 1671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:28.487326 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:28.488082 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:28.488374 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:28.488437 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:28.491254 (+ 2817us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:01:28.491255 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:01:28.491256 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:01:28.491263 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:01:28.492602 (+ 1339us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:01:28.493173 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:01:28.493174 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:01:28.493175 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:01:28.493238 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:01:28.493480 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:01:28.493481 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:01:28.493481 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:01:28.493603 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:01:28.493642 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0619 09:01:28.493983 (+ 341us) server_negotiation.cc:299] Negotiation successful
0619 09:01:28.494064 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260619 09:01:28.494148 7352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:28.483753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44591 (local address 127.0.98.60:39539)
0619 09:01:28.483793 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:28.483795 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:28.484224 (+ 429us) server_negotiation.cc:407] Connection header received
0619 09:01:28.485121 (+ 897us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:28.485122 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:28.485132 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:28.485192 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:01:28.485979 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:28.486752 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:28.488133 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:28.488417 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:28.488472 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:28.490784 (+ 2312us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:01:28.490785 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:01:28.490786 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:01:28.490793 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:01:28.492544 (+ 1751us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:01:28.493188 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:01:28.493189 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:01:28.493189 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:01:28.493245 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:01:28.493498 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:01:28.493499 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:01:28.493499 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:01:28.493623 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:01:28.493663 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0619 09:01:28.493981 (+ 318us) server_negotiation.cc:299] Negotiation successful
0619 09:01:28.494057 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260619 09:01:28.494653 7623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:28.483747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44403 (local address 127.0.98.2:36601)
0619 09:01:28.483910 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0619 09:01:28.483944 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0619 09:01:28.485013 (+ 1069us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:01:28.485283 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:01:28.485287 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:01:28.485680 (+ 393us) client_negotiation.cc:189] Negotiated authn=SASL
0619 09:01:28.485894 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:28.485899 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:28.487398 (+ 1499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:28.487399 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:28.488007 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:28.488009 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:28.488443 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:28.488444 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:28.488590 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:28.489071 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 09:01:28.489080 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0619 09:01:28.491154 (+ 2074us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 09:01:28.492686 (+ 1532us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:01:28.492687 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:01:28.492692 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:01:28.493092 (+ 400us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:01:28.493302 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:01:28.493303 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:01:28.493303 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:01:28.493415 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:01:28.493681 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 09:01:28.493686 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 09:01:28.493835 (+ 149us) client_negotiation.cc:769] Sending connection context
0619 09:01:28.494276 (+ 441us) client_negotiation.cc:240] Negotiation successful
0619 09:01:28.494434 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"mutex_wait_us":31,"thread_start_us":91,"threads_started":1}
I20260619 09:01:28.494995 7624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:28.483751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39539 (local address 127.0.98.2:44591)
0619 09:01:28.484142 (+ 391us) negotiation.cc:107] Waiting for socket to connect
0619 09:01:28.484151 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 09:01:28.485023 (+ 872us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:01:28.485305 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:01:28.485306 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:01:28.485668 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0619 09:01:28.485893 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:28.485897 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:28.486870 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:28.486870 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:28.488060 (+ 1190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:28.488062 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:28.488473 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:28.488474 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:28.488590 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:28.489109 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 09:01:28.489112 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 09:01:28.490697 (+ 1585us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 09:01:28.492679 (+ 1982us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:01:28.492685 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:01:28.492688 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:01:28.493094 (+ 406us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:01:28.493318 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:01:28.493318 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:01:28.493319 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:01:28.493422 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:01:28.493711 (+ 289us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 09:01:28.493712 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 09:01:28.493836 (+ 124us) client_negotiation.cc:769] Sending connection context
0619 09:01:28.494579 (+ 743us) client_negotiation.cc:240] Negotiation successful
0619 09:01:28.494676 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":371,"mutex_wait_us":52,"thread_start_us":91,"threads_started":1}
I20260619 09:01:28.495206 7619 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44403
I20260619 09:01:28.495292 7619 heartbeater.cc:464] Registering TS with master...
I20260619 09:01:28.495464 7619 heartbeater.cc:511] Master 127.0.98.62:44403 requested a full tablet report, sending...
I20260619 09:01:28.495486 7618 heartbeater.cc:347] Connected to a master server at 127.0.98.60:39539
I20260619 09:01:28.495517 7618 heartbeater.cc:464] Registering TS with master...
I20260619 09:01:28.495563 7618 heartbeater.cc:511] Master 127.0.98.60:39539 requested a full tablet report, sending...
I20260619 09:01:28.495821 6517 ts_manager.cc:194] Re-registered known tserver with Master: 00cca0af6ec34749baea4d021c20755e (127.0.98.2:37741)
I20260619 09:01:28.495849 7301 ts_manager.cc:194] Registered new tserver with Master: 00cca0af6ec34749baea4d021c20755e (127.0.98.2:37741)
I20260619 09:01:28.496253 6517 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:36601
I20260619 09:01:28.496682 7166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:28.484175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38701 (local address 127.0.98.61:34401)
0619 09:01:28.484214 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:28.484216 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:28.485230 (+ 1014us) server_negotiation.cc:407] Connection header received
0619 09:01:28.485317 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:28.485319 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:28.485330 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:28.485384 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:01:28.486003 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:28.486744 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:28.488521 (+ 1777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:28.488703 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:28.488755 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:28.492551 (+ 3796us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:01:28.492552 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:01:28.492552 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:01:28.492559 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:01:28.494059 (+ 1500us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:01:28.494426 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:01:28.494427 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:01:28.494427 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:01:28.494476 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:01:28.494644 (+ 168us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:01:28.494645 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:01:28.494646 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:01:28.494779 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:01:28.494836 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0619 09:01:28.496528 (+ 1692us) server_negotiation.cc:299] Negotiation successful
0619 09:01:28.496596 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260619 09:01:28.496694 7622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:28.484020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34401 (local address 127.0.98.2:38701)
0619 09:01:28.485177 (+ 1157us) negotiation.cc:107] Waiting for socket to connect
0619 09:01:28.485185 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0619 09:01:28.485253 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:01:28.485403 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:01:28.485404 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:01:28.485690 (+ 286us) client_negotiation.cc:189] Negotiated authn=SASL
0619 09:01:28.485894 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:28.485899 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:28.486870 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:28.486870 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:28.488062 (+ 1192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:28.488063 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:28.488767 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:28.488768 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:28.489127 (+ 359us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:28.489446 (+ 319us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 09:01:28.489449 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 09:01:28.492453 (+ 3004us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 09:01:28.494135 (+ 1682us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:01:28.494135 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:01:28.494136 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:01:28.494345 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:01:28.494535 (+ 190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:01:28.494535 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:01:28.494536 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:01:28.494585 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:01:28.495732 (+ 1147us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 09:01:28.495733 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 09:01:28.496430 (+ 697us) client_negotiation.cc:769] Sending connection context
0619 09:01:28.496493 (+ 63us) client_negotiation.cc:240] Negotiation successful
0619 09:01:28.496584 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1141,"mutex_wait_us":53,"thread_start_us":57,"threads_started":1}
I20260619 09:01:28.497078 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ez5NB/build/release/bin/kudu
/tmp/dist-test-task4ez5NB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859658134510-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859658134510-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859658134510-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859658134510-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:34875
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:44403
--webserver_port=46723
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859658134510-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42929
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:44403,127.0.98.61:34401
--tserver_master_addrs=127.0.98.62:44403,127.0.98.61:34401,127.0.98.60:39539 with env {KRB5CCNAME=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859658134510-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859658134510-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859658134510-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 09:01:28.498024 7620 heartbeater.cc:347] Connected to a master server at 127.0.98.61:34401
I20260619 09:01:28.498070 7620 heartbeater.cc:464] Registering TS with master...
I20260619 09:01:28.498129 7620 heartbeater.cc:511] Master 127.0.98.61:34401 requested a full tablet report, sending...
I20260619 09:01:28.498411 7115 ts_manager.cc:194] Registered new tserver with Master: 00cca0af6ec34749baea4d021c20755e (127.0.98.2:37741)
W20260619 09:01:28.574720 7625 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 09:01:28.574963 7625 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 09:01:28.574997 7625 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 09:01:28.576119 7625 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 09:01:28.576157 7625 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 09:01:28.576191 7625 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 09:01:28.577602 7625 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42929
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859658134510-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859658134510-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859658134510-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:34875
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859658134510-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=46723
--tserver_master_addrs=127.0.98.62:44403,127.0.98.61:34401,127.0.98.60:39539
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7625
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859658134510-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 09:01:28.577831 7625 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 09:01:28.578064 7625 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 09:01:28.583135 7630 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:01:28.583280 7633 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:01:28.583401 7631 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:01:28.583526 7625 server_base.cc:1034] running on GCE node
I20260619 09:01:28.583715 7625 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 09:01:28.583961 7625 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 09:01:28.585103 7625 hybrid_clock.cc:648] HybridClock initialized: now 1781859688585086 us; error 30 us; skew 500 ppm
Jun 19 09:01:28 dist-test-slave-mtdr krb5kdc[6472](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859688, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 09:01:28.587365 7625 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260619 09:01:28.587780 7625 webserver.cc:460] Webserver started at http://127.0.98.3:46723/ using document root <none> and password file <none>
I20260619 09:01:28.587986 7625 fs_manager.cc:359] Metadata directory not provided
I20260619 09:01:28.588047 7625 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 09:01:28.589298 7625 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 09:01:28.589787 7640 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 09:01:28.589948 7625 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260619 09:01:28.590025 7625 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859658134510-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859658134510-392-0/minicluster-data/ts-2/wal
uuid: "a5cc45b850424e7da92acf9089450d80"
format_stamp: "Formatted at 2026-06-19 09:01:27 on dist-test-slave-mtdr"
server_key: "80071be4a7876c182c9048c1acaceb93"
server_key_iv: "f3398164a79243eed432d8f88b31b9d5"
server_key_version: "encryptionkey@0"
I20260619 09:01:28.590370 7625 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859658134510-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859658134510-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859658134510-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 09:01:28.598796 7352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:28.596078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34934 (local address 127.0.98.60:39539)
0619 09:01:28.596133 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:28.596134 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:28.596145 (+ 11us) server_negotiation.cc:407] Connection header received
0619 09:01:28.596169 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:28.596170 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:28.596189 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:28.596244 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 09:01:28.596352 (+ 108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:28.596874 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:28.598136 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:28.598539 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:28.598602 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:28.598649 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0619 09:01:28.598714 (+ 65us) server_negotiation.cc:299] Negotiation successful
0619 09:01:28.598740 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260619 09:01:28.598843 7075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:28.596002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39539 (local address 127.0.0.1:34934)
0619 09:01:28.596050 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0619 09:01:28.596062 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 09:01:28.596115 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:01:28.596259 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:01:28.596260 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:01:28.596261 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 09:01:28.596304 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:28.596305 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:28.596936 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:28.596936 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:28.598040 (+ 1104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:28.598042 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:28.598605 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:28.598606 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:28.598659 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:28.598663 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 09:01:28.598708 (+ 45us) client_negotiation.cc:240] Negotiation successful
0619 09:01:28.598747 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260619 09:01:28.599176 7311 raft_consensus.cc:1203] T 00000000000000000000000000000000 P f1435fb0f12c416196a176e829751c3b [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260619 09:01:28.599335 7340 sys_catalog.cc:455] T 00000000000000000000000000000000 P f1435fb0f12c416196a176e829751c3b [sys.catalog]: SysCatalogTable state changed. Reason: New leader aaa8435622414a159f995f651c058e08. Latest consensus state: current_term: 1 leader_uuid: "aaa8435622414a159f995f651c058e08" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "aaa8435622414a159f995f651c058e08" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44403 } } peers { permanent_uuid: "13ea0fb4e8094bd8bda6763a0a07c2c5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34401 } attrs { promote: false } } peers { permanent_uuid: "f1435fb0f12c416196a176e829751c3b" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39539 } attrs { promote: true } } }
I20260619 09:01:28.599442 7340 sys_catalog.cc:458] T 00000000000000000000000000000000 P f1435fb0f12c416196a176e829751c3b [sys.catalog]: This master's current role is: LEARNER
I20260619 09:01:28.604698 7625 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 09:01:28.606361 7625 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 09:01:28.606479 7625 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 09:01:28.606712 7625 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 09:01:28.607029 7625 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 09:01:28.607062 7625 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 09:01:28.607084 7625 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 09:01:28.607108 7625 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 09:01:28.612293 7625 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34875
I20260619 09:01:28.612438 7753 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34875 every 8 connection(s)
I20260619 09:01:28.612659 7625 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859658134510-392-0/minicluster-data/ts-2/data/info.pb
Jun 19 09:01:28 dist-test-slave-mtdr krb5kdc[6472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859688, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
Jun 19 09:01:28 dist-test-slave-mtdr krb5kdc[6472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859688, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
Jun 19 09:01:28 dist-test-slave-mtdr krb5kdc[6472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859688, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260619 09:01:28.621299 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ez5NB/build/release/bin/kudu as pid 7625
I20260619 09:01:28.623557 7166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:28.613282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50709 (local address 127.0.98.61:34401)
0619 09:01:28.613323 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:28.613325 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:28.613577 (+ 252us) server_negotiation.cc:407] Connection header received
0619 09:01:28.614543 (+ 966us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:28.614545 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:28.614556 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:28.614602 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:01:28.615424 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:28.616160 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:28.617370 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:28.617548 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:28.617605 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:28.620594 (+ 2989us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:01:28.620595 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:01:28.620596 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:01:28.620603 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:01:28.622056 (+ 1453us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:01:28.622520 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:01:28.622521 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:01:28.622521 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:01:28.622574 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:01:28.622856 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:01:28.622857 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:01:28.622857 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:01:28.622983 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:01:28.623022 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0619 09:01:28.623399 (+ 377us) server_negotiation.cc:299] Negotiation successful
0619 09:01:28.623467 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260619 09:01:28.623754 7760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:28.613281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44403 (local address 127.0.98.3:44895)
0619 09:01:28.613664 (+ 383us) negotiation.cc:107] Waiting for socket to connect
0619 09:01:28.613673 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 09:01:28.614449 (+ 776us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:01:28.614673 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:01:28.614678 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:01:28.615110 (+ 432us) client_negotiation.cc:189] Negotiated authn=SASL
0619 09:01:28.615312 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:28.615316 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:28.617388 (+ 2072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:28.617389 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:28.618035 (+ 646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:28.618037 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:28.618437 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:28.618438 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:28.618657 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:28.619028 (+ 371us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 09:01:28.619032 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 09:01:28.620730 (+ 1698us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 09:01:28.622430 (+ 1700us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:01:28.622430 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:01:28.622431 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:01:28.622623 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:01:28.622856 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:01:28.622857 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:01:28.622857 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:01:28.622915 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:01:28.623189 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 09:01:28.623190 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 09:01:28.623242 (+ 52us) client_negotiation.cc:769] Sending connection context
0619 09:01:28.623354 (+ 112us) client_negotiation.cc:240] Negotiation successful
0619 09:01:28.623524 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":92,"threads_started":1}
I20260619 09:01:28.623888 7352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:28.613623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37031 (local address 127.0.98.60:39539)
0619 09:01:28.613662 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:28.613664 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:28.614650 (+ 986us) server_negotiation.cc:407] Connection header received
0619 09:01:28.614708 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:28.614709 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:28.614719 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:28.614764 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:01:28.616256 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:28.616709 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:28.617671 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:28.617928 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:28.618269 (+ 341us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:28.620287 (+ 2018us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:01:28.620288 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:01:28.620289 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:01:28.620306 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:01:28.622860 (+ 2554us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:01:28.623177 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:01:28.623177 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:01:28.623178 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:01:28.623228 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:01:28.623410 (+ 182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:01:28.623411 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:01:28.623411 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:01:28.623520 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:01:28.623556 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0619 09:01:28.623744 (+ 188us) server_negotiation.cc:299] Negotiation successful
0619 09:01:28.623805 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260619 09:01:28.623889 7759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:28.613281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34401 (local address 127.0.98.3:50709)
0619 09:01:28.613450 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0619 09:01:28.613492 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 09:01:28.614442 (+ 950us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:01:28.614673 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:01:28.614678 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:01:28.615126 (+ 448us) client_negotiation.cc:189] Negotiated authn=SASL
0619 09:01:28.615312 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:28.615318 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:28.616269 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:28.616270 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:28.617298 (+ 1028us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:28.617301 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:28.617619 (+ 318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:28.617620 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:28.617761 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:28.618286 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 09:01:28.618296 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 09:01:28.620523 (+ 2227us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 09:01:28.622142 (+ 1619us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:01:28.622149 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:01:28.622151 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:01:28.622432 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:01:28.622641 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:01:28.622642 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:01:28.622642 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:01:28.622760 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:01:28.623068 (+ 308us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 09:01:28.623074 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 09:01:28.623239 (+ 165us) client_negotiation.cc:769] Sending connection context
0619 09:01:28.623663 (+ 424us) client_negotiation.cc:240] Negotiation successful
0619 09:01:28.623753 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"mutex_wait_us":9,"thread_start_us":89,"threads_started":1}
I20260619 09:01:28.624228 7758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:28.613550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39539 (local address 127.0.98.3:37031)
0619 09:01:28.614600 (+ 1050us) negotiation.cc:107] Waiting for socket to connect
0619 09:01:28.614609 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 09:01:28.614655 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:01:28.614783 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:01:28.614784 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:01:28.615134 (+ 350us) client_negotiation.cc:189] Negotiated authn=SASL
0619 09:01:28.615312 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:28.615317 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:28.616804 (+ 1487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:28.616805 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:28.617581 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:28.617582 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:28.618280 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:28.618281 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:28.618342 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:28.618637 (+ 295us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 09:01:28.618642 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 09:01:28.620200 (+ 1558us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 09:01:28.622930 (+ 2730us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:01:28.622930 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:01:28.622931 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:01:28.623107 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:01:28.623293 (+ 186us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:01:28.623294 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:01:28.623294 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:01:28.623344 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:01:28.623599 (+ 255us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 09:01:28.623600 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 09:01:28.623634 (+ 34us) client_negotiation.cc:769] Sending connection context
0619 09:01:28.624054 (+ 420us) client_negotiation.cc:240] Negotiation successful
0619 09:01:28.624140 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1036,"mutex_wait_us":2,"thread_start_us":62,"threads_started":1}
I20260619 09:01:28.624394 7756 heartbeater.cc:347] Connected to a master server at 127.0.98.61:34401
I20260619 09:01:28.624456 7756 heartbeater.cc:464] Registering TS with master...
I20260619 09:01:28.624619 7756 heartbeater.cc:511] Master 127.0.98.61:34401 requested a full tablet report, sending...
I20260619 09:01:28.624749 6576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:28.613264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44895 (local address 127.0.98.62:44403)
0619 09:01:28.613314 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:28.613316 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:28.613728 (+ 412us) server_negotiation.cc:407] Connection header received
0619 09:01:28.614528 (+ 800us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:28.614529 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:28.614541 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:28.614597 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:01:28.615394 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:28.616165 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:28.618102 (+ 1937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:28.618371 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:28.618426 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:28.620974 (+ 2548us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:01:28.620975 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:01:28.620975 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:01:28.620984 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:01:28.622354 (+ 1370us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:01:28.622696 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:01:28.622697 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:01:28.622698 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:01:28.622751 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:01:28.622984 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:01:28.622985 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:01:28.622985 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:01:28.623110 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:01:28.623163 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0619 09:01:28.624596 (+ 1433us) server_negotiation.cc:299] Negotiation successful
0619 09:01:28.624666 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260619 09:01:28.625025 7754 heartbeater.cc:347] Connected to a master server at 127.0.98.60:39539
I20260619 09:01:28.625061 7755 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44403
I20260619 09:01:28.625106 7755 heartbeater.cc:464] Registering TS with master...
I20260619 09:01:28.625157 7755 heartbeater.cc:511] Master 127.0.98.62:44403 requested a full tablet report, sending...
I20260619 09:01:28.625161 7115 ts_manager.cc:194] Registered new tserver with Master: a5cc45b850424e7da92acf9089450d80 (127.0.98.3:34875)
I20260619 09:01:28.625069 7754 heartbeater.cc:464] Registering TS with master...
I20260619 09:01:28.625324 7754 heartbeater.cc:511] Master 127.0.98.60:39539 requested a full tablet report, sending...
I20260619 09:01:28.625479 6517 ts_manager.cc:194] Re-registered known tserver with Master: a5cc45b850424e7da92acf9089450d80 (127.0.98.3:34875)
I20260619 09:01:28.625726 7301 ts_manager.cc:194] Registered new tserver with Master: a5cc45b850424e7da92acf9089450d80 (127.0.98.3:34875)
I20260619 09:01:28.625917 6517 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:44895
I20260619 09:01:28.631436 7767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:28.622905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43276 (local address 127.0.98.61:34401)
0619 09:01:28.623621 (+ 716us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:28.623622 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:28.623742 (+ 120us) server_negotiation.cc:407] Connection header received
0619 09:01:28.623760 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:28.623761 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:28.623772 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:28.623821 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:01:28.624727 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:28.625721 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:28.627065 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:28.627313 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:28.627363 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:28.628578 (+ 1215us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:01:28.628579 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:01:28.628579 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:01:28.628585 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:01:28.630049 (+ 1464us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:01:28.630375 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:01:28.630376 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:01:28.630376 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:01:28.630426 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:01:28.630758 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:01:28.630759 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:01:28.630760 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:01:28.630904 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:01:28.631005 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0619 09:01:28.631263 (+ 258us) server_negotiation.cc:299] Negotiation successful
0619 09:01:28.631349 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":696,"thread_start_us":60,"threads_started":1}
I20260619 09:01:28.631470 6576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:28.622396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43070 (local address 127.0.98.62:44403)
0619 09:01:28.624837 (+ 2441us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:28.624839 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:28.624845 (+ 6us) server_negotiation.cc:407] Connection header received
0619 09:01:28.624862 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:28.624863 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:28.624871 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:28.624915 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:01:28.625566 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:28.626213 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:28.627168 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:28.627437 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:28.627507 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:28.628867 (+ 1360us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:01:28.628869 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:01:28.628869 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:01:28.628875 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:01:28.630356 (+ 1481us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:01:28.630857 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:01:28.630858 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:01:28.630858 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:01:28.630907 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:01:28.631063 (+ 156us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:01:28.631064 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:01:28.631065 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:01:28.631163 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:01:28.631254 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0619 09:01:28.631341 (+ 87us) server_negotiation.cc:299] Negotiation successful
0619 09:01:28.631408 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2427,"thread_start_us":72,"threads_started":1}
I20260619 09:01:28.633754 7769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:28.623218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34948 (local address 127.0.98.60:39539)
0619 09:01:28.623444 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:28.623446 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:28.626012 (+ 2566us) server_negotiation.cc:407] Connection header received
0619 09:01:28.626031 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:28.626032 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:28.626042 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:28.626087 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:01:28.627724 (+ 1637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:28.628364 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:28.629291 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:28.629762 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:28.629838 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:28.631363 (+ 1525us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:01:28.631364 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:01:28.631365 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:01:28.631371 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:01:28.632837 (+ 1466us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:01:28.633136 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:01:28.633137 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:01:28.633137 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:01:28.633196 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:01:28.633336 (+ 140us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:01:28.633336 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:01:28.633337 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:01:28.633460 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:01:28.633550 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0619 09:01:28.633604 (+ 54us) server_negotiation.cc:299] Negotiation successful
0619 09:01:28.633673 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":207,"thread_start_us":47,"threads_started":1}
I20260619 09:01:28.633324 6517 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43070:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 09:01:28.635056 6517 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 09:01:28.635685 7076 sys_catalog.cc:455] T 00000000000000000000000000000000 P aaa8435622414a159f995f651c058e08 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "aaa8435622414a159f995f651c058e08" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "aaa8435622414a159f995f651c058e08" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44403 } } peers { permanent_uuid: "13ea0fb4e8094bd8bda6763a0a07c2c5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34401 } attrs { promote: false } } peers { permanent_uuid: "f1435fb0f12c416196a176e829751c3b" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39539 } attrs { promote: true } } }
I20260619 09:01:28.635777 7076 sys_catalog.cc:458] T 00000000000000000000000000000000 P aaa8435622414a159f995f651c058e08 [sys.catalog]: This master's current role is: LEADER
I20260619 09:01:28.636854 7076 raft_consensus.cc:1050] T 00000000000000000000000000000000 P aaa8435622414a159f995f651c058e08: attempting to promote NON_VOTER f1435fb0f12c416196a176e829751c3b to VOTER
I20260619 09:01:28.636968 7076 consensus_queue.cc:237] T 00000000000000000000000000000000 P aaa8435622414a159f995f651c058e08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "aaa8435622414a159f995f651c058e08" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44403 } } peers { permanent_uuid: "13ea0fb4e8094bd8bda6763a0a07c2c5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34401 } attrs { promote: false } } peers { permanent_uuid: "f1435fb0f12c416196a176e829751c3b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39539 } attrs { promote: false } }
I20260619 09:01:28.637305 7125 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 13ea0fb4e8094bd8bda6763a0a07c2c5 [term 1 FOLLOWER]: Refusing update from remote peer aaa8435622414a159f995f651c058e08: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260619 09:01:28.637368 7311 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f1435fb0f12c416196a176e829751c3b [term 1 LEARNER]: Refusing update from remote peer aaa8435622414a159f995f651c058e08: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260619 09:01:28.637568 7772 consensus_queue.cc:1035] T 00000000000000000000000000000000 P aaa8435622414a159f995f651c058e08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1435fb0f12c416196a176e829751c3b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39539 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260619 09:01:28.637753 7773 consensus_queue.cc:1035] T 00000000000000000000000000000000 P aaa8435622414a159f995f651c058e08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "13ea0fb4e8094bd8bda6763a0a07c2c5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34401 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260619 09:01:28.638198 7772 raft_consensus.cc:2932] T 00000000000000000000000000000000 P aaa8435622414a159f995f651c058e08 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, f1435fb0f12c416196a176e829751c3b (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "aaa8435622414a159f995f651c058e08" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44403 } } peers { permanent_uuid: "13ea0fb4e8094bd8bda6763a0a07c2c5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34401 } attrs { promote: false } } peers { permanent_uuid: "f1435fb0f12c416196a176e829751c3b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39539 } attrs { promote: false } } }
I20260619 09:01:28.638331 7311 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f1435fb0f12c416196a176e829751c3b [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, f1435fb0f12c416196a176e829751c3b (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "aaa8435622414a159f995f651c058e08" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44403 } } peers { permanent_uuid: "13ea0fb4e8094bd8bda6763a0a07c2c5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34401 } attrs { promote: false } } peers { permanent_uuid: "f1435fb0f12c416196a176e829751c3b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39539 } attrs { promote: false } } }
I20260619 09:01:28.638458 7125 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 13ea0fb4e8094bd8bda6763a0a07c2c5 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, f1435fb0f12c416196a176e829751c3b (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "aaa8435622414a159f995f651c058e08" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44403 } } peers { permanent_uuid: "13ea0fb4e8094bd8bda6763a0a07c2c5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34401 } attrs { promote: false } } peers { permanent_uuid: "f1435fb0f12c416196a176e829751c3b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39539 } attrs { promote: false } } }
I20260619 09:01:28.639178 7772 sys_catalog.cc:455] T 00000000000000000000000000000000 P aaa8435622414a159f995f651c058e08 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "aaa8435622414a159f995f651c058e08" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "aaa8435622414a159f995f651c058e08" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44403 } } peers { permanent_uuid: "13ea0fb4e8094bd8bda6763a0a07c2c5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34401 } attrs { promote: false } } peers { permanent_uuid: "f1435fb0f12c416196a176e829751c3b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39539 } attrs { promote: false } } }
I20260619 09:01:28.639251 7772 sys_catalog.cc:458] T 00000000000000000000000000000000 P aaa8435622414a159f995f651c058e08 [sys.catalog]: This master's current role is: LEADER
I20260619 09:01:28.639227 7340 sys_catalog.cc:455] T 00000000000000000000000000000000 P f1435fb0f12c416196a176e829751c3b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "aaa8435622414a159f995f651c058e08" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "aaa8435622414a159f995f651c058e08" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44403 } } peers { permanent_uuid: "13ea0fb4e8094bd8bda6763a0a07c2c5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34401 } attrs { promote: false } } peers { permanent_uuid: "f1435fb0f12c416196a176e829751c3b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39539 } attrs { promote: false } } }
I20260619 09:01:28.639297 7340 sys_catalog.cc:458] T 00000000000000000000000000000000 P f1435fb0f12c416196a176e829751c3b [sys.catalog]: This master's current role is: FOLLOWER
I20260619 09:01:28.639319 7772 sys_catalog.cc:455] T 00000000000000000000000000000000 P aaa8435622414a159f995f651c058e08 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "aaa8435622414a159f995f651c058e08" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "aaa8435622414a159f995f651c058e08" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44403 } } peers { permanent_uuid: "13ea0fb4e8094bd8bda6763a0a07c2c5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34401 } attrs { promote: false } } peers { permanent_uuid: "f1435fb0f12c416196a176e829751c3b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39539 } attrs { promote: false } } }
I20260619 09:01:28.639364 7772 sys_catalog.cc:458] T 00000000000000000000000000000000 P aaa8435622414a159f995f651c058e08 [sys.catalog]: This master's current role is: LEADER
I20260619 09:01:28.639403 7772 sys_catalog.cc:455] T 00000000000000000000000000000000 P aaa8435622414a159f995f651c058e08 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "aaa8435622414a159f995f651c058e08" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "aaa8435622414a159f995f651c058e08" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44403 } } peers { permanent_uuid: "13ea0fb4e8094bd8bda6763a0a07c2c5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34401 } attrs { promote: false } } peers { permanent_uuid: "f1435fb0f12c416196a176e829751c3b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39539 } attrs { promote: false } } }
I20260619 09:01:28.639449 7772 sys_catalog.cc:458] T 00000000000000000000000000000000 P aaa8435622414a159f995f651c058e08 [sys.catalog]: This master's current role is: LEADER
I20260619 09:01:28.639999 7782 sys_catalog.cc:455] T 00000000000000000000000000000000 P 13ea0fb4e8094bd8bda6763a0a07c2c5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "aaa8435622414a159f995f651c058e08" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "aaa8435622414a159f995f651c058e08" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44403 } } peers { permanent_uuid: "13ea0fb4e8094bd8bda6763a0a07c2c5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34401 } attrs { promote: false } } peers { permanent_uuid: "f1435fb0f12c416196a176e829751c3b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39539 } attrs { promote: false } } }
I20260619 09:01:28.640084 7782 sys_catalog.cc:458] T 00000000000000000000000000000000 P 13ea0fb4e8094bd8bda6763a0a07c2c5 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 09:01:28.643573 7075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:28.639995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40909 (local address 127.0.0.1:51164)
0619 09:01:28.640115 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0619 09:01:28.640126 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 09:01:28.640165 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:01:28.640547 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:01:28.640548 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:01:28.640550 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 09:01:28.640603 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:28.640605 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:28.641548 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:28.641548 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:28.642550 (+ 1002us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:28.642552 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:28.643343 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:28.643344 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:28.643399 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:28.643402 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 09:01:28.643456 (+ 54us) client_negotiation.cc:240] Negotiation successful
0619 09:01:28.643491 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"mutex_wait_us":3}
I20260619 09:01:28.643582 7785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:28.640246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51164 (local address 127.0.98.1:40909)
0619 09:01:28.640388 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:28.640392 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:28.640403 (+ 11us) server_negotiation.cc:407] Connection header received
0619 09:01:28.640444 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:28.640445 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:28.640460 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:28.640545 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 09:01:28.640679 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:28.641466 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:28.642640 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:28.643276 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:28.643333 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:28.643409 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0619 09:01:28.643470 (+ 61us) server_negotiation.cc:299] Negotiation successful
0619 09:01:28.643504 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":69,"threads_started":1}
I20260619 09:01:28.643914 7783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:28.640060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34875 (local address 127.0.0.1:38098)
0619 09:01:28.640180 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0619 09:01:28.640190 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 09:01:28.640238 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:01:28.640567 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:01:28.640568 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:01:28.640570 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 09:01:28.640621 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:28.640623 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:28.641837 (+ 1214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:28.641838 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:28.642896 (+ 1058us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:28.642898 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:28.643702 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:28.643704 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:28.643766 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:28.643770 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 09:01:28.643797 (+ 27us) client_negotiation.cc:240] Negotiation successful
0619 09:01:28.643847 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":83,"threads_started":1}
I20260619 09:01:28.644021 7784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:28.640093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43742 (local address 127.0.98.2:37741)
0619 09:01:28.640346 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:28.640350 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:28.640538 (+ 188us) server_negotiation.cc:407] Connection header received
0619 09:01:28.640670 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:28.640673 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:28.640690 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:28.640764 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 09:01:28.640856 (+ 92us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:28.642042 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:28.642898 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:28.643616 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:28.643672 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:28.643744 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0619 09:01:28.643905 (+ 161us) server_negotiation.cc:299] Negotiation successful
0619 09:01:28.643945 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":220,"thread_start_us":86,"threads_started":1}
I20260619 09:01:28.644214 7787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:28.639996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37741 (local address 127.0.0.1:43742)
0619 09:01:28.640478 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0619 09:01:28.640486 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0619 09:01:28.640561 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:01:28.640768 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:01:28.640768 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:01:28.640770 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 09:01:28.640801 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:28.640802 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:28.642108 (+ 1306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:28.642109 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:28.642791 (+ 682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:28.642793 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:28.643772 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:28.643773 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:28.643822 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:28.643825 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 09:01:28.644109 (+ 284us) client_negotiation.cc:240] Negotiation successful
0619 09:01:28.644149 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":468,"mutex_wait_us":366,"thread_start_us":61,"threads_started":1}
I20260619 09:01:28.644294 7786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:28.640262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38098 (local address 127.0.98.3:34875)
0619 09:01:28.640418 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:28.640422 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:28.640429 (+ 7us) server_negotiation.cc:407] Connection header received
0619 09:01:28.640462 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:28.640464 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:28.640477 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:28.640691 (+ 214us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 09:01:28.640744 (+ 53us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:28.641753 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:28.642965 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:28.643602 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:28.644068 (+ 466us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:28.644159 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0619 09:01:28.644183 (+ 24us) server_negotiation.cc:299] Negotiation successful
0619 09:01:28.644223 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":68,"threads_started":1}
I20260619 09:01:28.644287 7416 tablet_service.cc:1467] Processing CreateTablet for tablet e4d18f185fd843bc925f70f48526568f (DEFAULT_TABLE table=test-table [id=3137678c350e4b1bb6ef11866ec36405]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 09:01:28.644609 7416 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4d18f185fd843bc925f70f48526568f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 09:01:28.644842 7552 tablet_service.cc:1467] Processing CreateTablet for tablet e4d18f185fd843bc925f70f48526568f (DEFAULT_TABLE table=test-table [id=3137678c350e4b1bb6ef11866ec36405]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 09:01:28.644843 7688 tablet_service.cc:1467] Processing CreateTablet for tablet e4d18f185fd843bc925f70f48526568f (DEFAULT_TABLE table=test-table [id=3137678c350e4b1bb6ef11866ec36405]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 09:01:28.645151 7552 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4d18f185fd843bc925f70f48526568f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 09:01:28.645695 7688 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4d18f185fd843bc925f70f48526568f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 09:01:28.647569 7788 tablet_bootstrap.cc:492] T e4d18f185fd843bc925f70f48526568f P 551feb9174794395a270058ce5ed45c5: Bootstrap starting.
I20260619 09:01:28.648146 7788 tablet_bootstrap.cc:654] T e4d18f185fd843bc925f70f48526568f P 551feb9174794395a270058ce5ed45c5: Neither blocks nor log segments found. Creating new log.
I20260619 09:01:28.648427 7788 log.cc:824] T e4d18f185fd843bc925f70f48526568f P 551feb9174794395a270058ce5ed45c5: Log is configured to *not* fsync() on all Append() calls
I20260619 09:01:28.648629 7789 tablet_bootstrap.cc:492] T e4d18f185fd843bc925f70f48526568f P 00cca0af6ec34749baea4d021c20755e: Bootstrap starting.
I20260619 09:01:28.648873 7791 tablet_bootstrap.cc:492] T e4d18f185fd843bc925f70f48526568f P a5cc45b850424e7da92acf9089450d80: Bootstrap starting.
I20260619 09:01:28.649118 7788 tablet_bootstrap.cc:492] T e4d18f185fd843bc925f70f48526568f P 551feb9174794395a270058ce5ed45c5: No bootstrap required, opened a new log
I20260619 09:01:28.649186 7788 ts_tablet_manager.cc:1397] T e4d18f185fd843bc925f70f48526568f P 551feb9174794395a270058ce5ed45c5: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 09:01:28.649353 7791 tablet_bootstrap.cc:654] T e4d18f185fd843bc925f70f48526568f P a5cc45b850424e7da92acf9089450d80: Neither blocks nor log segments found. Creating new log.
I20260619 09:01:28.649369 7789 tablet_bootstrap.cc:654] T e4d18f185fd843bc925f70f48526568f P 00cca0af6ec34749baea4d021c20755e: Neither blocks nor log segments found. Creating new log.
I20260619 09:01:28.649605 7791 log.cc:824] T e4d18f185fd843bc925f70f48526568f P a5cc45b850424e7da92acf9089450d80: Log is configured to *not* fsync() on all Append() calls
I20260619 09:01:28.649634 7789 log.cc:824] T e4d18f185fd843bc925f70f48526568f P 00cca0af6ec34749baea4d021c20755e: Log is configured to *not* fsync() on all Append() calls
I20260619 09:01:28.650216 7791 tablet_bootstrap.cc:492] T e4d18f185fd843bc925f70f48526568f P a5cc45b850424e7da92acf9089450d80: No bootstrap required, opened a new log
I20260619 09:01:28.650270 7791 ts_tablet_manager.cc:1397] T e4d18f185fd843bc925f70f48526568f P a5cc45b850424e7da92acf9089450d80: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260619 09:01:28.651127 7789 tablet_bootstrap.cc:492] T e4d18f185fd843bc925f70f48526568f P 00cca0af6ec34749baea4d021c20755e: No bootstrap required, opened a new log
I20260619 09:01:28.651187 7789 ts_tablet_manager.cc:1397] T e4d18f185fd843bc925f70f48526568f P 00cca0af6ec34749baea4d021c20755e: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260619 09:01:28.651923 7788 raft_consensus.cc:348] T e4d18f185fd843bc925f70f48526568f P 551feb9174794395a270058ce5ed45c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "551feb9174794395a270058ce5ed45c5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40909 } } peers { permanent_uuid: "00cca0af6ec34749baea4d021c20755e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37741 } } peers { permanent_uuid: "a5cc45b850424e7da92acf9089450d80" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34875 } }
I20260619 09:01:28.652091 7788 raft_consensus.cc:374] T e4d18f185fd843bc925f70f48526568f P 551feb9174794395a270058ce5ed45c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 09:01:28.652127 7788 raft_consensus.cc:729] T e4d18f185fd843bc925f70f48526568f P 551feb9174794395a270058ce5ed45c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 551feb9174794395a270058ce5ed45c5, State: Initialized, Role: FOLLOWER
I20260619 09:01:28.652267 7788 consensus_queue.cc:260] T e4d18f185fd843bc925f70f48526568f P 551feb9174794395a270058ce5ed45c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "551feb9174794395a270058ce5ed45c5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40909 } } peers { permanent_uuid: "00cca0af6ec34749baea4d021c20755e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37741 } } peers { permanent_uuid: "a5cc45b850424e7da92acf9089450d80" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34875 } }
I20260619 09:01:28.652510 7788 ts_tablet_manager.cc:1428] T e4d18f185fd843bc925f70f48526568f P 551feb9174794395a270058ce5ed45c5: Time spent starting tablet: real 0.003s user 0.000s sys 0.004s
I20260619 09:01:28.652665 7483 heartbeater.cc:503] Master 127.0.98.62:44403 was elected leader, sending a full tablet report...
I20260619 09:01:28.652597 7791 raft_consensus.cc:348] T e4d18f185fd843bc925f70f48526568f P a5cc45b850424e7da92acf9089450d80 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "551feb9174794395a270058ce5ed45c5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40909 } } peers { permanent_uuid: "00cca0af6ec34749baea4d021c20755e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37741 } } peers { permanent_uuid: "a5cc45b850424e7da92acf9089450d80" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34875 } }
I20260619 09:01:28.652734 7791 raft_consensus.cc:374] T e4d18f185fd843bc925f70f48526568f P a5cc45b850424e7da92acf9089450d80 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 09:01:28.652767 7791 raft_consensus.cc:729] T e4d18f185fd843bc925f70f48526568f P a5cc45b850424e7da92acf9089450d80 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5cc45b850424e7da92acf9089450d80, State: Initialized, Role: FOLLOWER
I20260619 09:01:28.652896 7791 consensus_queue.cc:260] T e4d18f185fd843bc925f70f48526568f P a5cc45b850424e7da92acf9089450d80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "551feb9174794395a270058ce5ed45c5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40909 } } peers { permanent_uuid: "00cca0af6ec34749baea4d021c20755e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37741 } } peers { permanent_uuid: "a5cc45b850424e7da92acf9089450d80" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34875 } }
I20260619 09:01:28.653141 7791 ts_tablet_manager.cc:1428] T e4d18f185fd843bc925f70f48526568f P a5cc45b850424e7da92acf9089450d80: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260619 09:01:28.653287 7755 heartbeater.cc:503] Master 127.0.98.62:44403 was elected leader, sending a full tablet report...
I20260619 09:01:28.653046 7789 raft_consensus.cc:348] T e4d18f185fd843bc925f70f48526568f P 00cca0af6ec34749baea4d021c20755e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "551feb9174794395a270058ce5ed45c5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40909 } } peers { permanent_uuid: "00cca0af6ec34749baea4d021c20755e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37741 } } peers { permanent_uuid: "a5cc45b850424e7da92acf9089450d80" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34875 } }
I20260619 09:01:28.654139 7789 raft_consensus.cc:374] T e4d18f185fd843bc925f70f48526568f P 00cca0af6ec34749baea4d021c20755e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 09:01:28.654410 7789 raft_consensus.cc:729] T e4d18f185fd843bc925f70f48526568f P 00cca0af6ec34749baea4d021c20755e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00cca0af6ec34749baea4d021c20755e, State: Initialized, Role: FOLLOWER
I20260619 09:01:28.654605 7789 consensus_queue.cc:260] T e4d18f185fd843bc925f70f48526568f P 00cca0af6ec34749baea4d021c20755e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "551feb9174794395a270058ce5ed45c5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40909 } } peers { permanent_uuid: "00cca0af6ec34749baea4d021c20755e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37741 } } peers { permanent_uuid: "a5cc45b850424e7da92acf9089450d80" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34875 } }
I20260619 09:01:28.654901 7619 heartbeater.cc:503] Master 127.0.98.62:44403 was elected leader, sending a full tablet report...
I20260619 09:01:28.654899 7789 ts_tablet_manager.cc:1428] T e4d18f185fd843bc925f70f48526568f P 00cca0af6ec34749baea4d021c20755e: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
W20260619 09:01:28.733994 7621 tablet.cc:2369] T e4d18f185fd843bc925f70f48526568f P 00cca0af6ec34749baea4d021c20755e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 09:01:28.826246 7795 raft_consensus.cc:482] T e4d18f185fd843bc925f70f48526568f P a5cc45b850424e7da92acf9089450d80 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 09:01:28.826380 7795 raft_consensus.cc:504] T e4d18f185fd843bc925f70f48526568f P a5cc45b850424e7da92acf9089450d80 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "551feb9174794395a270058ce5ed45c5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40909 } } peers { permanent_uuid: "00cca0af6ec34749baea4d021c20755e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37741 } } peers { permanent_uuid: "a5cc45b850424e7da92acf9089450d80" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34875 } }
I20260619 09:01:28.826795 7795 leader_election.cc:288] T e4d18f185fd843bc925f70f48526568f P a5cc45b850424e7da92acf9089450d80 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 551feb9174794395a270058ce5ed45c5 (127.0.98.1:40909), 00cca0af6ec34749baea4d021c20755e (127.0.98.2:37741)
I20260619 09:01:28.829556 7785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:28.827086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35143 (local address 127.0.98.1:40909)
0619 09:01:28.827125 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:28.827127 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:28.827156 (+ 29us) server_negotiation.cc:407] Connection header received
0619 09:01:28.827182 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:28.827183 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:28.827193 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:28.827235 (+ 42us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 09:01:28.827425 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:28.827939 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:28.828880 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:28.829239 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:28.829295 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:28.829306 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0619 09:01:28.829450 (+ 144us) server_negotiation.cc:299] Negotiation successful
0619 09:01:28.829488 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260619 09:01:28.829558 7784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:28.827205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39443 (local address 127.0.98.2:37741)
0619 09:01:28.827280 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:28.827282 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:28.827289 (+ 7us) server_negotiation.cc:407] Connection header received
0619 09:01:28.827314 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:28.827315 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:28.827329 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:28.827392 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 09:01:28.827534 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:28.827984 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:28.828889 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:28.829246 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:28.829315 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:28.829325 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0619 09:01:28.829455 (+ 130us) server_negotiation.cc:299] Negotiation successful
0619 09:01:28.829488 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 09:01:28.829595 7759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:28.826980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40909 (local address 127.0.98.3:35143)
0619 09:01:28.827064 (+ 84us) negotiation.cc:107] Waiting for socket to connect
0619 09:01:28.827075 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 09:01:28.827119 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:01:28.827272 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:01:28.827273 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:01:28.827275 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 09:01:28.827354 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:28.827356 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:28.828015 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:28.828016 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:28.828780 (+ 764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:28.828783 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:28.829312 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:28.829313 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:28.829384 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:28.829387 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 09:01:28.829439 (+ 52us) client_negotiation.cc:240] Negotiation successful
0619 09:01:28.829480 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260619 09:01:28.829607 7758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:28.827012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37741 (local address 127.0.98.3:39443)
0619 09:01:28.827090 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0619 09:01:28.827100 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 09:01:28.827141 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:01:28.827404 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:01:28.827405 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:01:28.827406 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 09:01:28.827466 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:28.827468 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:28.828072 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:28.828073 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:28.828782 (+ 709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:28.828784 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:28.829325 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:28.829326 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:28.829393 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:28.829396 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 09:01:28.829442 (+ 46us) client_negotiation.cc:240] Negotiation successful
0619 09:01:28.829482 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62,"mutex_wait_us":1}
I20260619 09:01:28.829952 7436 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4d18f185fd843bc925f70f48526568f" candidate_uuid: "a5cc45b850424e7da92acf9089450d80" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "551feb9174794395a270058ce5ed45c5" is_pre_election: true
I20260619 09:01:28.830125 7436 raft_consensus.cc:2454] T e4d18f185fd843bc925f70f48526568f P 551feb9174794395a270058ce5ed45c5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a5cc45b850424e7da92acf9089450d80 in term 0.
I20260619 09:01:28.830120 7572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4d18f185fd843bc925f70f48526568f" candidate_uuid: "a5cc45b850424e7da92acf9089450d80" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00cca0af6ec34749baea4d021c20755e" is_pre_election: true
I20260619 09:01:28.830273 7572 raft_consensus.cc:2454] T e4d18f185fd843bc925f70f48526568f P 00cca0af6ec34749baea4d021c20755e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a5cc45b850424e7da92acf9089450d80 in term 0.
I20260619 09:01:28.830407 7643 leader_election.cc:302] T e4d18f185fd843bc925f70f48526568f P a5cc45b850424e7da92acf9089450d80 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 551feb9174794395a270058ce5ed45c5, a5cc45b850424e7da92acf9089450d80; no voters:
I20260619 09:01:28.830541 7795 raft_consensus.cc:2781] T e4d18f185fd843bc925f70f48526568f P a5cc45b850424e7da92acf9089450d80 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 09:01:28.830595 7795 raft_consensus.cc:482] T e4d18f185fd843bc925f70f48526568f P a5cc45b850424e7da92acf9089450d80 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 09:01:28.830618 7795 raft_consensus.cc:3037] T e4d18f185fd843bc925f70f48526568f P a5cc45b850424e7da92acf9089450d80 [term 0 FOLLOWER]: Advancing to term 1
I20260619 09:01:28.831313 7795 raft_consensus.cc:504] T e4d18f185fd843bc925f70f48526568f P a5cc45b850424e7da92acf9089450d80 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "551feb9174794395a270058ce5ed45c5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40909 } } peers { permanent_uuid: "00cca0af6ec34749baea4d021c20755e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37741 } } peers { permanent_uuid: "a5cc45b850424e7da92acf9089450d80" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34875 } }
I20260619 09:01:28.831434 7795 leader_election.cc:288] T e4d18f185fd843bc925f70f48526568f P a5cc45b850424e7da92acf9089450d80 [CANDIDATE]: Term 1 election: Requested vote from peers 551feb9174794395a270058ce5ed45c5 (127.0.98.1:40909), 00cca0af6ec34749baea4d021c20755e (127.0.98.2:37741)
I20260619 09:01:28.831568 7572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4d18f185fd843bc925f70f48526568f" candidate_uuid: "a5cc45b850424e7da92acf9089450d80" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00cca0af6ec34749baea4d021c20755e"
I20260619 09:01:28.831568 7436 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4d18f185fd843bc925f70f48526568f" candidate_uuid: "a5cc45b850424e7da92acf9089450d80" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "551feb9174794395a270058ce5ed45c5"
I20260619 09:01:28.831636 7436 raft_consensus.cc:3037] T e4d18f185fd843bc925f70f48526568f P 551feb9174794395a270058ce5ed45c5 [term 0 FOLLOWER]: Advancing to term 1
I20260619 09:01:28.831637 7572 raft_consensus.cc:3037] T e4d18f185fd843bc925f70f48526568f P 00cca0af6ec34749baea4d021c20755e [term 0 FOLLOWER]: Advancing to term 1
I20260619 09:01:28.832289 7572 raft_consensus.cc:2454] T e4d18f185fd843bc925f70f48526568f P 00cca0af6ec34749baea4d021c20755e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5cc45b850424e7da92acf9089450d80 in term 1.
I20260619 09:01:28.832293 7436 raft_consensus.cc:2454] T e4d18f185fd843bc925f70f48526568f P 551feb9174794395a270058ce5ed45c5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5cc45b850424e7da92acf9089450d80 in term 1.
I20260619 09:01:28.832445 7643 leader_election.cc:302] T e4d18f185fd843bc925f70f48526568f P a5cc45b850424e7da92acf9089450d80 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 551feb9174794395a270058ce5ed45c5, a5cc45b850424e7da92acf9089450d80; no voters:
I20260619 09:01:28.832518 7795 raft_consensus.cc:2781] T e4d18f185fd843bc925f70f48526568f P a5cc45b850424e7da92acf9089450d80 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 09:01:28.832659 7795 raft_consensus.cc:686] T e4d18f185fd843bc925f70f48526568f P a5cc45b850424e7da92acf9089450d80 [term 1 LEADER]: Becoming Leader. State: Replica: a5cc45b850424e7da92acf9089450d80, State: Running, Role: LEADER
I20260619 09:01:28.832743 7795 consensus_queue.cc:237] T e4d18f185fd843bc925f70f48526568f P a5cc45b850424e7da92acf9089450d80 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "551feb9174794395a270058ce5ed45c5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40909 } } peers { permanent_uuid: "00cca0af6ec34749baea4d021c20755e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37741 } } peers { permanent_uuid: "a5cc45b850424e7da92acf9089450d80" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34875 } }
I20260619 09:01:28.833573 6517 catalog_manager.cc:5495] T e4d18f185fd843bc925f70f48526568f P a5cc45b850424e7da92acf9089450d80 reported cstate change: term changed from 0 to 1, leader changed from <none> to a5cc45b850424e7da92acf9089450d80 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "a5cc45b850424e7da92acf9089450d80" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "551feb9174794395a270058ce5ed45c5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40909 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "00cca0af6ec34749baea4d021c20755e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37741 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a5cc45b850424e7da92acf9089450d80" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34875 } health_report { overall_health: HEALTHY } } }
W20260619 09:01:28.846928 7485 tablet.cc:2369] T e4d18f185fd843bc925f70f48526568f P 551feb9174794395a270058ce5ed45c5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 09:01:28.863320 7757 tablet.cc:2369] T e4d18f185fd843bc925f70f48526568f P a5cc45b850424e7da92acf9089450d80: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 09:01:28.867695 7786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:28.865501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38104 (local address 127.0.98.3:34875)
0619 09:01:28.865546 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:28.865548 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:28.865556 (+ 8us) server_negotiation.cc:407] Connection header received
0619 09:01:28.865588 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:28.865588 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:28.865607 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:28.865671 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 09:01:28.865768 (+ 97us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:28.866246 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:28.867045 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:28.867225 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:28.867274 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:28.867451 (+ 177us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 09:01:28.867550 (+ 99us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 09:01:28.867604 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0619 09:01:28.867621 (+ 17us) server_negotiation.cc:299] Negotiation successful
0619 09:01:28.867637 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260619 09:01:28.869355 7436 raft_consensus.cc:1261] T e4d18f185fd843bc925f70f48526568f P 551feb9174794395a270058ce5ed45c5 [term 1 FOLLOWER]: Refusing update from remote peer a5cc45b850424e7da92acf9089450d80: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 09:01:28.869433 7572 raft_consensus.cc:1261] T e4d18f185fd843bc925f70f48526568f P 00cca0af6ec34749baea4d021c20755e [term 1 FOLLOWER]: Refusing update from remote peer a5cc45b850424e7da92acf9089450d80: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 09:01:28.869694 7795 consensus_queue.cc:1035] T e4d18f185fd843bc925f70f48526568f P a5cc45b850424e7da92acf9089450d80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "551feb9174794395a270058ce5ed45c5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40909 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 09:01:28.869840 7797 consensus_queue.cc:1035] T e4d18f185fd843bc925f70f48526568f P a5cc45b850424e7da92acf9089450d80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00cca0af6ec34749baea4d021c20755e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37741 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 09:01:28.872228 7800 mvcc.cc:205] Tried to move back new op lower bound from 7298497285605478400 to 7298497285458960384. Current Snapshot: MvccSnapshot[applied={T|T < 7298497285605478400}]
I20260619 09:01:28.874392 6517 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43070:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 09:01:28.874487 6517 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43070:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 09:01:28.874498 7802 mvcc.cc:205] Tried to move back new op lower bound from 7298497285605478400 to 7298497285458960384. Current Snapshot: MvccSnapshot[applied={T|T < 7298497285605478400 or (T in {7298497285605478400})}]
I20260619 09:01:28.876186 6517 catalog_manager.cc:5782] T 00000000000000000000000000000000 P aaa8435622414a159f995f651c058e08: Sending DeleteTablet for 3 replicas of tablet e4d18f185fd843bc925f70f48526568f
I20260619 09:01:28.876534 7416 tablet_service.cc:1514] Processing DeleteTablet for tablet e4d18f185fd843bc925f70f48526568f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 09:01:28 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51164
I20260619 09:01:28.876561 7552 tablet_service.cc:1514] Processing DeleteTablet for tablet e4d18f185fd843bc925f70f48526568f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 09:01:28 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43742
I20260619 09:01:28.876703 7810 tablet_replica.cc:330] T e4d18f185fd843bc925f70f48526568f P 551feb9174794395a270058ce5ed45c5: stopping tablet replica
I20260619 09:01:28.876840 7810 raft_consensus.cc:2229] T e4d18f185fd843bc925f70f48526568f P 551feb9174794395a270058ce5ed45c5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 09:01:28.876933 7810 raft_consensus.cc:2258] T e4d18f185fd843bc925f70f48526568f P 551feb9174794395a270058ce5ed45c5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 09:01:28.877279 7688 tablet_service.cc:1514] Processing DeleteTablet for tablet e4d18f185fd843bc925f70f48526568f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 09:01:28 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38098
I20260619 09:01:28.877544 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ez5NB/build/release/bin/kudu with pid 6485
I20260619 09:01:28.877275 7687 tablet_service.cc:1514] Processing DeleteTablet for tablet e4d18f185fd843bc925f70f48526568f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 09:01:28 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38098
I20260619 09:01:28.879033 7812 tablet_replica.cc:330] T e4d18f185fd843bc925f70f48526568f P a5cc45b850424e7da92acf9089450d80: stopping tablet replica
I20260619 09:01:28.879132 7812 raft_consensus.cc:2229] T e4d18f185fd843bc925f70f48526568f P a5cc45b850424e7da92acf9089450d80 [term 1 LEADER]: Raft consensus shutting down.
I20260619 09:01:28.879221 7812 raft_consensus.cc:2258] T e4d18f185fd843bc925f70f48526568f P a5cc45b850424e7da92acf9089450d80 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 09:01:28.879581 7812 ts_tablet_manager.cc:1905] T e4d18f185fd843bc925f70f48526568f P a5cc45b850424e7da92acf9089450d80: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 09:01:28.881408 7812 ts_tablet_manager.cc:1918] T e4d18f185fd843bc925f70f48526568f P a5cc45b850424e7da92acf9089450d80: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 09:01:28.881464 7812 log.cc:1196] T e4d18f185fd843bc925f70f48526568f P a5cc45b850424e7da92acf9089450d80: Deleting WAL directory at /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859658134510-392-0/minicluster-data/ts-2/wal/wals/e4d18f185fd843bc925f70f48526568f
I20260619 09:01:28.881664 7812 ts_tablet_manager.cc:1939] T e4d18f185fd843bc925f70f48526568f P a5cc45b850424e7da92acf9089450d80: Deleting consensus metadata
I20260619 09:01:28.883213 7811 tablet_replica.cc:330] T e4d18f185fd843bc925f70f48526568f P 00cca0af6ec34749baea4d021c20755e: stopping tablet replica
I20260619 09:01:28.883354 7811 raft_consensus.cc:2229] T e4d18f185fd843bc925f70f48526568f P 00cca0af6ec34749baea4d021c20755e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 09:01:28.883484 7811 raft_consensus.cc:2258] T e4d18f185fd843bc925f70f48526568f P 00cca0af6ec34749baea4d021c20755e [term 1 FOLLOWER]: Raft consensus is shut down!
W20260619 09:01:28.883947 7644 connection.cc:535] server connection from 127.0.0.1:38098 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260619 09:01:28.884159 7483 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:44403 (0 consecutive failures): Network error: Failed to send heartbeat to master: recv got EOF from 127.0.98.62:44403 (error 108)
I20260619 09:01:28.884428 7811 ts_tablet_manager.cc:1905] T e4d18f185fd843bc925f70f48526568f P 00cca0af6ec34749baea4d021c20755e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 09:01:28.885988 7811 ts_tablet_manager.cc:1918] T e4d18f185fd843bc925f70f48526568f P 00cca0af6ec34749baea4d021c20755e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 09:01:28.886063 7811 log.cc:1196] T e4d18f185fd843bc925f70f48526568f P 00cca0af6ec34749baea4d021c20755e: Deleting WAL directory at /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859658134510-392-0/minicluster-data/ts-1/wal/wals/e4d18f185fd843bc925f70f48526568f
I20260619 09:01:28.886325 7811 ts_tablet_manager.cc:1939] T e4d18f185fd843bc925f70f48526568f P 00cca0af6ec34749baea4d021c20755e: Deleting consensus metadata
I20260619 09:01:28.886991 7810 ts_tablet_manager.cc:1905] T e4d18f185fd843bc925f70f48526568f P 551feb9174794395a270058ce5ed45c5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 09:01:28.888516 7810 ts_tablet_manager.cc:1918] T e4d18f185fd843bc925f70f48526568f P 551feb9174794395a270058ce5ed45c5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 09:01:28.888590 7810 log.cc:1196] T e4d18f185fd843bc925f70f48526568f P 551feb9174794395a270058ce5ed45c5: Deleting WAL directory at /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859658134510-392-0/minicluster-data/ts-0/wal/wals/e4d18f185fd843bc925f70f48526568f
I20260619 09:01:28.888820 7810 ts_tablet_manager.cc:1939] T e4d18f185fd843bc925f70f48526568f P 551feb9174794395a270058ce5ed45c5: Deleting consensus metadata
W20260619 09:01:28.889137 7371 connection.cc:441] server connection from 127.0.0.1:51164 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51164 (request call id 1) could send its response
W20260619 09:01:28.891173 7505 connection.cc:441] server connection from 127.0.0.1:43742 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43742 (request call id 1) could send its response
I20260619 09:01:28.893622 7767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:28.886117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43284 (local address 127.0.98.61:34401)
0619 09:01:28.886155 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:28.886157 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:28.886322 (+ 165us) server_negotiation.cc:407] Connection header received
0619 09:01:28.886477 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:28.886478 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:28.886494 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:28.886521 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:01:28.887308 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:28.888043 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:28.889306 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:28.889521 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:28.889579 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:28.890857 (+ 1278us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:01:28.890858 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:01:28.890859 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:01:28.890867 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:01:28.892464 (+ 1597us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:01:28.892823 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:01:28.892824 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:01:28.892824 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:01:28.892878 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:01:28.893080 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:01:28.893082 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:01:28.893082 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:01:28.893247 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:01:28.893345 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0619 09:01:28.893445 (+ 100us) server_negotiation.cc:299] Negotiation successful
0619 09:01:28.893527 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260619 09:01:28.894045 7769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:28.886368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34952 (local address 127.0.98.60:39539)
0619 09:01:28.886405 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:28.886406 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:28.886598 (+ 192us) server_negotiation.cc:407] Connection header received
0619 09:01:28.886847 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:28.886848 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:28.886863 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:28.886896 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:01:28.887585 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:28.888376 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:28.889422 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:28.889745 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:28.889786 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:28.891126 (+ 1340us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:01:28.891127 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:01:28.891127 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:01:28.891134 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:01:28.892751 (+ 1617us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:01:28.893236 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:01:28.893237 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:01:28.893237 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:01:28.893289 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:01:28.893468 (+ 179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:01:28.893469 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:01:28.893469 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:01:28.893590 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:01:28.893647 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0619 09:01:28.893878 (+ 231us) server_negotiation.cc:299] Negotiation successful
0619 09:01:28.893960 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260619 09:01:29.874630 7819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:29.874385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44403 (local address 127.0.98.2:57109)
0619 09:01:29.874503 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0619 09:01:29.874578 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":56,"threads_started":1}
W20260619 09:01:29.874838 7619 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:44403 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:44403: connect: Connection refused (error 111)
I20260619 09:01:29.883530 7820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:29.883308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44403 (local address 127.0.98.3:49651)
0619 09:01:29.883419 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0619 09:01:29.883485 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":56,"threads_started":1}
W20260619 09:01:29.883733 7755 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:44403 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:44403: connect: Connection refused (error 111)
I20260619 09:01:29.884636 7821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:29.884446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44403 (local address 127.0.98.1:58211)
0619 09:01:29.884545 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0619 09:01:29.884602 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84,"thread_start_us":45,"threads_started":1}
I20260619 09:01:30.376202 7822 raft_consensus.cc:482] T 00000000000000000000000000000000 P 13ea0fb4e8094bd8bda6763a0a07c2c5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader aaa8435622414a159f995f651c058e08)
I20260619 09:01:30.376348 7822 raft_consensus.cc:504] T 00000000000000000000000000000000 P 13ea0fb4e8094bd8bda6763a0a07c2c5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "aaa8435622414a159f995f651c058e08" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44403 } } peers { permanent_uuid: "13ea0fb4e8094bd8bda6763a0a07c2c5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34401 } attrs { promote: false } } peers { permanent_uuid: "f1435fb0f12c416196a176e829751c3b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39539 } attrs { promote: false } }
I20260619 09:01:30.376709 7822 leader_election.cc:288] T 00000000000000000000000000000000 P 13ea0fb4e8094bd8bda6763a0a07c2c5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers aaa8435622414a159f995f651c058e08 (127.0.98.62:44403), f1435fb0f12c416196a176e829751c3b (127.0.98.60:39539)
I20260619 09:01:30.377064 7823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:30.376841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44403 (local address 127.0.0.1:43580)
0619 09:01:30.376975 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0619 09:01:30.377021 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":49,"threads_started":1}
I20260619 09:01:30.377055 7825 raft_consensus.cc:482] T 00000000000000000000000000000000 P f1435fb0f12c416196a176e829751c3b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader aaa8435622414a159f995f651c058e08)
W20260619 09:01:30.377178 7101 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44403: connect: Connection refused (error 111)
I20260619 09:01:30.377164 7825 raft_consensus.cc:504] T 00000000000000000000000000000000 P f1435fb0f12c416196a176e829751c3b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "aaa8435622414a159f995f651c058e08" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44403 } } peers { permanent_uuid: "13ea0fb4e8094bd8bda6763a0a07c2c5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34401 } attrs { promote: false } } peers { permanent_uuid: "f1435fb0f12c416196a176e829751c3b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39539 } attrs { promote: false } }
I20260619 09:01:30.377584 7823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:30.377495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44403 (local address 127.0.0.1:43606)
0619 09:01:30.377535 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0619 09:01:30.377558 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260619 09:01:30.377660 7101 leader_election.cc:334] T 00000000000000000000000000000000 P 13ea0fb4e8094bd8bda6763a0a07c2c5 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer aaa8435622414a159f995f651c058e08 (127.0.98.62:44403): Network error: Client connection negotiation failed: client connection to 127.0.98.62:44403: connect: Connection refused (error 111)
I20260619 09:01:30.377753 7828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:30.377458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44403 (local address 127.0.0.1:43590)
0619 09:01:30.377545 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0619 09:01:30.377596 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74,"thread_start_us":43,"threads_started":1}
I20260619 09:01:30.377837 7825 leader_election.cc:288] T 00000000000000000000000000000000 P f1435fb0f12c416196a176e829751c3b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers aaa8435622414a159f995f651c058e08 (127.0.98.62:44403), 13ea0fb4e8094bd8bda6763a0a07c2c5 (127.0.98.61:34401)
W20260619 09:01:30.377892 7287 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44403: connect: Connection refused (error 111)
I20260619 09:01:30.378252 7828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:30.378185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44403 (local address 127.0.0.1:43614)
0619 09:01:30.378218 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0619 09:01:30.378235 (+ 17us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
W20260619 09:01:30.378309 7287 leader_election.cc:334] T 00000000000000000000000000000000 P f1435fb0f12c416196a176e829751c3b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer aaa8435622414a159f995f651c058e08 (127.0.98.62:44403): Network error: Client connection negotiation failed: client connection to 127.0.98.62:44403: connect: Connection refused (error 111)
I20260619 09:01:30.380187 7826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:30.376984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34956 (local address 127.0.98.60:39539)
0619 09:01:30.377089 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:30.377091 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:30.377105 (+ 14us) server_negotiation.cc:407] Connection header received
0619 09:01:30.377131 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:30.377132 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:30.377143 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:30.377188 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 09:01:30.377407 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:30.378113 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:30.379349 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:30.379918 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:30.379987 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:30.379999 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0619 09:01:30.380114 (+ 115us) server_negotiation.cc:299] Negotiation successful
0619 09:01:30.380146 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80,"thread_start_us":54,"threads_started":1}
I20260619 09:01:30.380265 7824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:30.376857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39539 (local address 127.0.0.1:34956)
0619 09:01:30.377007 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0619 09:01:30.377017 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 09:01:30.377078 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:01:30.377272 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:01:30.377273 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:01:30.377275 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 09:01:30.377331 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:30.377333 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:30.378208 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:30.378209 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:30.379277 (+ 1068us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:30.379278 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:30.379996 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:30.379997 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:30.380046 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:30.380050 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 09:01:30.380105 (+ 55us) client_negotiation.cc:240] Negotiation successful
0619 09:01:30.380147 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":42,"threads_started":1}
I20260619 09:01:30.380481 7311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "13ea0fb4e8094bd8bda6763a0a07c2c5" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "f1435fb0f12c416196a176e829751c3b" is_pre_election: true
I20260619 09:01:30.380570 7311 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f1435fb0f12c416196a176e829751c3b [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 13ea0fb4e8094bd8bda6763a0a07c2c5 in term 1.
I20260619 09:01:30.380704 7102 leader_election.cc:302] T 00000000000000000000000000000000 P 13ea0fb4e8094bd8bda6763a0a07c2c5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 13ea0fb4e8094bd8bda6763a0a07c2c5, f1435fb0f12c416196a176e829751c3b; no voters: aaa8435622414a159f995f651c058e08
I20260619 09:01:30.380815 7822 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 13ea0fb4e8094bd8bda6763a0a07c2c5 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260619 09:01:30.380846 7822 raft_consensus.cc:482] T 00000000000000000000000000000000 P 13ea0fb4e8094bd8bda6763a0a07c2c5 [term 1 FOLLOWER]: Starting leader election (detected failure of leader aaa8435622414a159f995f651c058e08)
I20260619 09:01:30.380864 7822 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 13ea0fb4e8094bd8bda6763a0a07c2c5 [term 1 FOLLOWER]: Advancing to term 2
I20260619 09:01:30.380889 7829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:30.377625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34401 (local address 127.0.0.1:43290)
0619 09:01:30.377783 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0619 09:01:30.377793 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 09:01:30.377842 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:01:30.378025 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:01:30.378025 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:01:30.378027 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 09:01:30.378067 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:30.378070 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:30.378979 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:30.378980 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:30.380019 (+ 1039us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:30.380021 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:30.380664 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:30.380665 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:30.380739 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:30.380742 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 09:01:30.380787 (+ 45us) client_negotiation.cc:240] Negotiation successful
0619 09:01:30.380824 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":48,"threads_started":1}
I20260619 09:01:30.380937 7830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:30.377741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43290 (local address 127.0.98.61:34401)
0619 09:01:30.377926 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:30.377928 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:30.377934 (+ 6us) server_negotiation.cc:407] Connection header received
0619 09:01:30.377950 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:30.377951 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:30.377957 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:30.378009 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 09:01:30.378149 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:30.378913 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:30.380082 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:30.380570 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:30.380625 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:30.380635 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0619 09:01:30.380827 (+ 192us) server_negotiation.cc:299] Negotiation successful
0619 09:01:30.380864 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":76,"threads_started":1}
I20260619 09:01:30.381582 7822 raft_consensus.cc:504] T 00000000000000000000000000000000 P 13ea0fb4e8094bd8bda6763a0a07c2c5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "aaa8435622414a159f995f651c058e08" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44403 } } peers { permanent_uuid: "13ea0fb4e8094bd8bda6763a0a07c2c5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34401 } attrs { promote: false } } peers { permanent_uuid: "f1435fb0f12c416196a176e829751c3b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39539 } attrs { promote: false } }
I20260619 09:01:30.381690 7822 leader_election.cc:288] T 00000000000000000000000000000000 P 13ea0fb4e8094bd8bda6763a0a07c2c5 [CANDIDATE]: Term 2 election: Requested vote from peers aaa8435622414a159f995f651c058e08 (127.0.98.62:44403), f1435fb0f12c416196a176e829751c3b (127.0.98.60:39539)
I20260619 09:01:30.381860 7824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:30.381764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44403 (local address 127.0.0.1:43618)
0619 09:01:30.381812 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0619 09:01:30.381837 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260619 09:01:30.381871 7311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "13ea0fb4e8094bd8bda6763a0a07c2c5" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "f1435fb0f12c416196a176e829751c3b"
I20260619 09:01:30.381878 7125 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f1435fb0f12c416196a176e829751c3b" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "13ea0fb4e8094bd8bda6763a0a07c2c5" is_pre_election: true
I20260619 09:01:30.381928 7311 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f1435fb0f12c416196a176e829751c3b [term 1 FOLLOWER]: Advancing to term 2
I20260619 09:01:30.381971 7125 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 13ea0fb4e8094bd8bda6763a0a07c2c5 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f1435fb0f12c416196a176e829751c3b in current term 2: Already voted for candidate 13ea0fb4e8094bd8bda6763a0a07c2c5 in this term.
I20260619 09:01:30.382138 7288 leader_election.cc:302] T 00000000000000000000000000000000 P f1435fb0f12c416196a176e829751c3b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f1435fb0f12c416196a176e829751c3b; no voters: 13ea0fb4e8094bd8bda6763a0a07c2c5, aaa8435622414a159f995f651c058e08
I20260619 09:01:30.382334 7824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:30.382112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44403 (local address 127.0.0.1:43624)
0619 09:01:30.382288 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0619 09:01:30.382314 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161}
W20260619 09:01:30.382407 7101 leader_election.cc:334] T 00000000000000000000000000000000 P 13ea0fb4e8094bd8bda6763a0a07c2c5 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer aaa8435622414a159f995f651c058e08 (127.0.98.62:44403): Network error: Client connection negotiation failed: client connection to 127.0.98.62:44403: connect: Connection refused (error 111)
I20260619 09:01:30.382643 7311 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f1435fb0f12c416196a176e829751c3b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 13ea0fb4e8094bd8bda6763a0a07c2c5 in term 2.
I20260619 09:01:30.382730 7825 raft_consensus.cc:2726] T 00000000000000000000000000000000 P f1435fb0f12c416196a176e829751c3b [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 09:01:30.382789 7102 leader_election.cc:302] T 00000000000000000000000000000000 P 13ea0fb4e8094bd8bda6763a0a07c2c5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 13ea0fb4e8094bd8bda6763a0a07c2c5, f1435fb0f12c416196a176e829751c3b; no voters: aaa8435622414a159f995f651c058e08
I20260619 09:01:30.382880 7822 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 13ea0fb4e8094bd8bda6763a0a07c2c5 [term 2 FOLLOWER]: Leader election won for term 2
I20260619 09:01:30.383006 7822 raft_consensus.cc:686] T 00000000000000000000000000000000 P 13ea0fb4e8094bd8bda6763a0a07c2c5 [term 2 LEADER]: Becoming Leader. State: Replica: 13ea0fb4e8094bd8bda6763a0a07c2c5, State: Running, Role: LEADER
I20260619 09:01:30.383069 7822 consensus_queue.cc:237] T 00000000000000000000000000000000 P 13ea0fb4e8094bd8bda6763a0a07c2c5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "aaa8435622414a159f995f651c058e08" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44403 } } peers { permanent_uuid: "13ea0fb4e8094bd8bda6763a0a07c2c5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34401 } attrs { promote: false } } peers { permanent_uuid: "f1435fb0f12c416196a176e829751c3b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39539 } attrs { promote: false } }
I20260619 09:01:30.383370 7832 sys_catalog.cc:455] T 00000000000000000000000000000000 P 13ea0fb4e8094bd8bda6763a0a07c2c5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 13ea0fb4e8094bd8bda6763a0a07c2c5. Latest consensus state: current_term: 2 leader_uuid: "13ea0fb4e8094bd8bda6763a0a07c2c5" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "aaa8435622414a159f995f651c058e08" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44403 } } peers { permanent_uuid: "13ea0fb4e8094bd8bda6763a0a07c2c5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34401 } attrs { promote: false } } peers { permanent_uuid: "f1435fb0f12c416196a176e829751c3b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39539 } attrs { promote: false } } }
I20260619 09:01:30.383452 7832 sys_catalog.cc:458] T 00000000000000000000000000000000 P 13ea0fb4e8094bd8bda6763a0a07c2c5 [sys.catalog]: This master's current role is: LEADER
I20260619 09:01:30.383610 7834 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 09:01:30.384035 7834 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 09:01:30.384091 7834 catalog_manager.cc:1259] Loaded cluster ID: aac0526a83f348348927f89c7ec729d5
I20260619 09:01:30.384111 7834 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 09:01:30.384227 7834 catalog_manager.cc:1504] Loading token signing keys...
I20260619 09:01:30.384299 7834 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 13ea0fb4e8094bd8bda6763a0a07c2c5: Loaded TSK: 0
I20260619 09:01:30.384398 7834 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 09:01:30.433666 7114 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43284:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 09:01:30.435344 7114 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 09:01:30.436286 7824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:30.436183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44403 (local address 127.0.0.1:43654)
0619 09:01:30.436231 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0619 09:01:30.436261 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260619 09:01:30.436343 7311 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f1435fb0f12c416196a176e829751c3b [term 2 FOLLOWER]: Refusing update from remote peer 13ea0fb4e8094bd8bda6763a0a07c2c5: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260619 09:01:30.436584 7824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:30.436501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44403 (local address 127.0.0.1:43668)
0619 09:01:30.436538 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0619 09:01:30.436562 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
I20260619 09:01:30.436641 7832 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 13ea0fb4e8094bd8bda6763a0a07c2c5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1435fb0f12c416196a176e829751c3b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39539 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260619 09:01:30.436770 7101 consensus_peers.cc:487] T 00000000000000000000000000000000 P 13ea0fb4e8094bd8bda6763a0a07c2c5 -> Peer aaa8435622414a159f995f651c058e08 (127.0.98.62:44403): Couldn't send request to peer aaa8435622414a159f995f651c058e08. Status: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44403: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260619 09:01:30.437249 7825 sys_catalog.cc:455] T 00000000000000000000000000000000 P f1435fb0f12c416196a176e829751c3b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 13ea0fb4e8094bd8bda6763a0a07c2c5. Latest consensus state: current_term: 2 leader_uuid: "13ea0fb4e8094bd8bda6763a0a07c2c5" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "aaa8435622414a159f995f651c058e08" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44403 } } peers { permanent_uuid: "13ea0fb4e8094bd8bda6763a0a07c2c5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34401 } attrs { promote: false } } peers { permanent_uuid: "f1435fb0f12c416196a176e829751c3b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39539 } attrs { promote: false } } }
I20260619 09:01:30.437397 7825 sys_catalog.cc:458] T 00000000000000000000000000000000 P f1435fb0f12c416196a176e829751c3b [sys.catalog]: This master's current role is: FOLLOWER
I20260619 09:01:30.437676 7835 mvcc.cc:205] Tried to move back new op lower bound from 7298497292025348096 to 7298497291808321536. Current Snapshot: MvccSnapshot[applied={T|T < 7298497292025348096}]
I20260619 09:01:30.437788 7822 sys_catalog.cc:455] T 00000000000000000000000000000000 P 13ea0fb4e8094bd8bda6763a0a07c2c5 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "13ea0fb4e8094bd8bda6763a0a07c2c5" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "aaa8435622414a159f995f651c058e08" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44403 } } peers { permanent_uuid: "13ea0fb4e8094bd8bda6763a0a07c2c5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34401 } attrs { promote: false } } peers { permanent_uuid: "f1435fb0f12c416196a176e829751c3b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39539 } attrs { promote: false } } }
I20260619 09:01:30.437870 7822 sys_catalog.cc:458] T 00000000000000000000000000000000 P 13ea0fb4e8094bd8bda6763a0a07c2c5 [sys.catalog]: This master's current role is: LEADER
I20260619 09:01:30.437923 7825 sys_catalog.cc:455] T 00000000000000000000000000000000 P f1435fb0f12c416196a176e829751c3b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "13ea0fb4e8094bd8bda6763a0a07c2c5" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "aaa8435622414a159f995f651c058e08" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44403 } } peers { permanent_uuid: "13ea0fb4e8094bd8bda6763a0a07c2c5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34401 } attrs { promote: false } } peers { permanent_uuid: "f1435fb0f12c416196a176e829751c3b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39539 } attrs { promote: false } } }
I20260619 09:01:30.438009 7825 sys_catalog.cc:458] T 00000000000000000000000000000000 P f1435fb0f12c416196a176e829751c3b [sys.catalog]: This master's current role is: FOLLOWER
I20260619 09:01:30.438225 7836 mvcc.cc:205] Tried to move back new op lower bound from 7298497292025348096 to 7298497291808321536. Current Snapshot: MvccSnapshot[applied={T|T < 7298497292025348096}]
I20260619 09:01:30.442540 7843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:30.439724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43744 (local address 127.0.98.2:37741)
0619 09:01:30.440022 (+ 298us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:30.440024 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:30.440031 (+ 7us) server_negotiation.cc:407] Connection header received
0619 09:01:30.440053 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:30.440054 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:30.440063 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:30.440115 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 09:01:30.440217 (+ 102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:30.440934 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:30.441818 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:30.442254 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:30.442318 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:30.442329 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0619 09:01:30.442443 (+ 114us) server_negotiation.cc:299] Negotiation successful
0619 09:01:30.442475 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":278,"thread_start_us":71,"threads_started":1}
I20260619 09:01:30.442539 7824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:30.439555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37741 (local address 127.0.0.1:43744)
0619 09:01:30.439738 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0619 09:01:30.439746 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0619 09:01:30.439774 (+ 28us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:01:30.440129 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:01:30.440130 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:01:30.440131 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 09:01:30.440162 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:30.440163 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:30.441000 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:30.441001 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:30.441745 (+ 744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:30.441747 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:30.442323 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:30.442324 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:30.442383 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:30.442387 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 09:01:30.442433 (+ 46us) client_negotiation.cc:240] Negotiation successful
0619 09:01:30.442471 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":72,"threads_started":1}
I20260619 09:01:30.442725 7823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:30.439489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40909 (local address 127.0.0.1:51176)
0619 09:01:30.439528 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0619 09:01:30.439538 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 09:01:30.439582 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:01:30.439774 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:01:30.439775 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:01:30.439778 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 09:01:30.439850 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:30.439852 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:30.440596 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:30.440597 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:30.441654 (+ 1057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:30.441656 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:30.442535 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:30.442536 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:30.442596 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:30.442599 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 09:01:30.442626 (+ 27us) client_negotiation.cc:240] Negotiation successful
0619 09:01:30.442662 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25}
I20260619 09:01:30.442847 7841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:30.439500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34875 (local address 127.0.0.1:38106)
0619 09:01:30.439663 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0619 09:01:30.439674 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 09:01:30.439721 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:01:30.439968 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:01:30.439969 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:01:30.439970 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 09:01:30.440013 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:30.440015 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:30.440890 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:30.440891 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:30.441930 (+ 1039us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:30.441932 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:30.442634 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:30.442634 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:30.442681 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:30.442684 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 09:01:30.442723 (+ 39us) client_negotiation.cc:240] Negotiation successful
0619 09:01:30.442754 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144}
I20260619 09:01:30.442950 7552 tablet_service.cc:1467] Processing CreateTablet for tablet 44c9c858d29a4610803799759d801474 (DEFAULT_TABLE table=test-table [id=3ee55b69707e4bbcba49751c8411319f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 09:01:30.443001 7840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:30.439545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51176 (local address 127.0.98.1:40909)
0619 09:01:30.439646 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:30.439648 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:30.439656 (+ 8us) server_negotiation.cc:407] Connection header received
0619 09:01:30.439680 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:30.439681 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:30.439692 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:30.439911 (+ 219us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 09:01:30.439946 (+ 35us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:30.440531 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:30.441857 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:30.442436 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:30.442854 (+ 418us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:30.442870 (+ 16us) server_negotiation.cc:1035] Waiting for connection context
0619 09:01:30.442890 (+ 20us) server_negotiation.cc:299] Negotiation successful
0619 09:01:30.442930 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81,"thread_start_us":51,"threads_started":1}
I20260619 09:01:30.443084 7552 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44c9c858d29a4610803799759d801474. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 09:01:30.443084 7842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:30.439653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38106 (local address 127.0.98.3:34875)
0619 09:01:30.439855 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:30.439857 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:30.439864 (+ 7us) server_negotiation.cc:407] Connection header received
0619 09:01:30.439888 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:30.439889 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:30.439899 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:30.439952 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 09:01:30.440086 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:30.440815 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:30.442008 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:30.442542 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:30.442595 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:30.442607 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0619 09:01:30.442976 (+ 369us) server_negotiation.cc:299] Negotiation successful
0619 09:01:30.443020 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":182,"thread_start_us":56,"threads_started":1}
I20260619 09:01:30.443215 7416 tablet_service.cc:1467] Processing CreateTablet for tablet 44c9c858d29a4610803799759d801474 (DEFAULT_TABLE table=test-table [id=3ee55b69707e4bbcba49751c8411319f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 09:01:30.443323 7416 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44c9c858d29a4610803799759d801474. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 09:01:30.444231 7687 tablet_service.cc:1467] Processing CreateTablet for tablet 44c9c858d29a4610803799759d801474 (DEFAULT_TABLE table=test-table [id=3ee55b69707e4bbcba49751c8411319f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 09:01:30.444345 7687 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44c9c858d29a4610803799759d801474. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 09:01:30.444514 7845 tablet_bootstrap.cc:492] T 44c9c858d29a4610803799759d801474 P 551feb9174794395a270058ce5ed45c5: Bootstrap starting.
I20260619 09:01:30.444833 7845 tablet_bootstrap.cc:654] T 44c9c858d29a4610803799759d801474 P 551feb9174794395a270058ce5ed45c5: Neither blocks nor log segments found. Creating new log.
I20260619 09:01:30.445386 7845 tablet_bootstrap.cc:492] T 44c9c858d29a4610803799759d801474 P 551feb9174794395a270058ce5ed45c5: No bootstrap required, opened a new log
I20260619 09:01:30.445438 7845 ts_tablet_manager.cc:1397] T 44c9c858d29a4610803799759d801474 P 551feb9174794395a270058ce5ed45c5: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260619 09:01:30.445569 7845 raft_consensus.cc:348] T 44c9c858d29a4610803799759d801474 P 551feb9174794395a270058ce5ed45c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "551feb9174794395a270058ce5ed45c5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40909 } } peers { permanent_uuid: "a5cc45b850424e7da92acf9089450d80" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34875 } } peers { permanent_uuid: "00cca0af6ec34749baea4d021c20755e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37741 } }
I20260619 09:01:30.445643 7845 raft_consensus.cc:374] T 44c9c858d29a4610803799759d801474 P 551feb9174794395a270058ce5ed45c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 09:01:30.445672 7845 raft_consensus.cc:729] T 44c9c858d29a4610803799759d801474 P 551feb9174794395a270058ce5ed45c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 551feb9174794395a270058ce5ed45c5, State: Initialized, Role: FOLLOWER
I20260619 09:01:30.445719 7845 consensus_queue.cc:260] T 44c9c858d29a4610803799759d801474 P 551feb9174794395a270058ce5ed45c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "551feb9174794395a270058ce5ed45c5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40909 } } peers { permanent_uuid: "a5cc45b850424e7da92acf9089450d80" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34875 } } peers { permanent_uuid: "00cca0af6ec34749baea4d021c20755e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37741 } }
I20260619 09:01:30.445757 7847 tablet_bootstrap.cc:492] T 44c9c858d29a4610803799759d801474 P a5cc45b850424e7da92acf9089450d80: Bootstrap starting.
I20260619 09:01:30.445854 7845 ts_tablet_manager.cc:1428] T 44c9c858d29a4610803799759d801474 P 551feb9174794395a270058ce5ed45c5: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260619 09:01:30.446070 7847 tablet_bootstrap.cc:654] T 44c9c858d29a4610803799759d801474 P a5cc45b850424e7da92acf9089450d80: Neither blocks nor log segments found. Creating new log.
I20260619 09:01:30.446164 7844 tablet_bootstrap.cc:492] T 44c9c858d29a4610803799759d801474 P 00cca0af6ec34749baea4d021c20755e: Bootstrap starting.
I20260619 09:01:30.446513 7844 tablet_bootstrap.cc:654] T 44c9c858d29a4610803799759d801474 P 00cca0af6ec34749baea4d021c20755e: Neither blocks nor log segments found. Creating new log.
I20260619 09:01:30.446707 7484 heartbeater.cc:503] Master 127.0.98.61:34401 was elected leader, sending a full tablet report...
I20260619 09:01:30.446889 7849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:30.446054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44403 (local address 127.0.98.1:38927)
0619 09:01:30.446799 (+ 745us) negotiation.cc:107] Waiting for socket to connect
0619 09:01:30.446851 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":715,"thread_start_us":73,"threads_started":1}
I20260619 09:01:30.447180 7844 tablet_bootstrap.cc:492] T 44c9c858d29a4610803799759d801474 P 00cca0af6ec34749baea4d021c20755e: No bootstrap required, opened a new log
I20260619 09:01:30.447232 7844 ts_tablet_manager.cc:1397] T 44c9c858d29a4610803799759d801474 P 00cca0af6ec34749baea4d021c20755e: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260619 09:01:30.447268 7849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:30.447182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44403 (local address 127.0.98.1:43595)
0619 09:01:30.447214 (+ 32us) negotiation.cc:107] Waiting for socket to connect
0619 09:01:30.447244 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
I20260619 09:01:30.447369 7844 raft_consensus.cc:348] T 44c9c858d29a4610803799759d801474 P 00cca0af6ec34749baea4d021c20755e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "551feb9174794395a270058ce5ed45c5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40909 } } peers { permanent_uuid: "a5cc45b850424e7da92acf9089450d80" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34875 } } peers { permanent_uuid: "00cca0af6ec34749baea4d021c20755e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37741 } }
I20260619 09:01:30.447441 7844 raft_consensus.cc:374] T 44c9c858d29a4610803799759d801474 P 00cca0af6ec34749baea4d021c20755e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 09:01:30.447461 7844 raft_consensus.cc:729] T 44c9c858d29a4610803799759d801474 P 00cca0af6ec34749baea4d021c20755e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00cca0af6ec34749baea4d021c20755e, State: Initialized, Role: FOLLOWER
I20260619 09:01:30.447507 7844 consensus_queue.cc:260] T 44c9c858d29a4610803799759d801474 P 00cca0af6ec34749baea4d021c20755e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "551feb9174794395a270058ce5ed45c5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40909 } } peers { permanent_uuid: "a5cc45b850424e7da92acf9089450d80" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34875 } } peers { permanent_uuid: "00cca0af6ec34749baea4d021c20755e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37741 } }
I20260619 09:01:30.447644 7844 ts_tablet_manager.cc:1428] T 44c9c858d29a4610803799759d801474 P 00cca0af6ec34749baea4d021c20755e: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260619 09:01:30.447933 7847 tablet_bootstrap.cc:492] T 44c9c858d29a4610803799759d801474 P a5cc45b850424e7da92acf9089450d80: No bootstrap required, opened a new log
I20260619 09:01:30.447984 7847 ts_tablet_manager.cc:1397] T 44c9c858d29a4610803799759d801474 P a5cc45b850424e7da92acf9089450d80: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 09:01:30.448042 7620 heartbeater.cc:503] Master 127.0.98.61:34401 was elected leader, sending a full tablet report...
I20260619 09:01:30.448104 7847 raft_consensus.cc:348] T 44c9c858d29a4610803799759d801474 P a5cc45b850424e7da92acf9089450d80 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "551feb9174794395a270058ce5ed45c5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40909 } } peers { permanent_uuid: "a5cc45b850424e7da92acf9089450d80" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34875 } } peers { permanent_uuid: "00cca0af6ec34749baea4d021c20755e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37741 } }
I20260619 09:01:30.448139 7853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:30.447795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44403 (local address 127.0.98.2:32935)
0619 09:01:30.448062 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0619 09:01:30.448100 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":73,"threads_started":1}
I20260619 09:01:30.448166 7847 raft_consensus.cc:374] T 44c9c858d29a4610803799759d801474 P a5cc45b850424e7da92acf9089450d80 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 09:01:30.448184 7847 raft_consensus.cc:729] T 44c9c858d29a4610803799759d801474 P a5cc45b850424e7da92acf9089450d80 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5cc45b850424e7da92acf9089450d80, State: Initialized, Role: FOLLOWER
I20260619 09:01:30.448220 7847 consensus_queue.cc:260] T 44c9c858d29a4610803799759d801474 P a5cc45b850424e7da92acf9089450d80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "551feb9174794395a270058ce5ed45c5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40909 } } peers { permanent_uuid: "a5cc45b850424e7da92acf9089450d80" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34875 } } peers { permanent_uuid: "00cca0af6ec34749baea4d021c20755e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37741 } }
I20260619 09:01:30.448339 7847 ts_tablet_manager.cc:1428] T 44c9c858d29a4610803799759d801474 P a5cc45b850424e7da92acf9089450d80: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260619 09:01:30.448665 7856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:30.448499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44403 (local address 127.0.98.3:59079)
0619 09:01:30.448603 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0619 09:01:30.448634 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88,"thread_start_us":58,"threads_started":1}
I20260619 09:01:30.448771 7853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:30.448456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44403 (local address 127.0.98.2:56555)
0619 09:01:30.448723 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0619 09:01:30.448747 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":67,"threads_started":1}
I20260619 09:01:30.448812 7756 heartbeater.cc:503] Master 127.0.98.61:34401 was elected leader, sending a full tablet report...
I20260619 09:01:30.449030 7856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:30.448961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44403 (local address 127.0.98.3:54577)
0619 09:01:30.448992 (+ 31us) negotiation.cc:107] Waiting for socket to connect
0619 09:01:30.449014 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
I20260619 09:01:30.684185 7848 raft_consensus.cc:482] T 44c9c858d29a4610803799759d801474 P 551feb9174794395a270058ce5ed45c5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 09:01:30.684293 7848 raft_consensus.cc:504] T 44c9c858d29a4610803799759d801474 P 551feb9174794395a270058ce5ed45c5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "551feb9174794395a270058ce5ed45c5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40909 } } peers { permanent_uuid: "a5cc45b850424e7da92acf9089450d80" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34875 } } peers { permanent_uuid: "00cca0af6ec34749baea4d021c20755e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37741 } }
I20260619 09:01:30.684695 7848 leader_election.cc:288] T 44c9c858d29a4610803799759d801474 P 551feb9174794395a270058ce5ed45c5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a5cc45b850424e7da92acf9089450d80 (127.0.98.3:34875), 00cca0af6ec34749baea4d021c20755e (127.0.98.2:37741)
I20260619 09:01:30.688103 7842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:30.684965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48313 (local address 127.0.98.3:34875)
0619 09:01:30.685096 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:30.685098 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:30.685284 (+ 186us) server_negotiation.cc:407] Connection header received
0619 09:01:30.685346 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:30.685347 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:30.685358 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:30.685413 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 09:01:30.685548 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:30.685993 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:30.687424 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:30.687778 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:30.687843 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:30.687853 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0619 09:01:30.688005 (+ 152us) server_negotiation.cc:299] Negotiation successful
0619 09:01:30.688043 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
I20260619 09:01:30.688097 7849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:30.684875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37741 (local address 127.0.98.1:40311)
0619 09:01:30.684987 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0619 09:01:30.684999 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 09:01:30.685070 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:01:30.685258 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:01:30.685259 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:01:30.685262 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 09:01:30.685341 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:30.685343 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:30.686001 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:30.686002 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:30.687319 (+ 1317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:30.687321 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:30.687825 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:30.687826 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:30.687910 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:30.687913 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 09:01:30.687961 (+ 48us) client_negotiation.cc:240] Negotiation successful
0619 09:01:30.688008 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":89,"mutex_wait_us":53}
I20260619 09:01:30.688103 7843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:30.684954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40311 (local address 127.0.98.2:37741)
0619 09:01:30.685077 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:30.685079 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:30.685090 (+ 11us) server_negotiation.cc:407] Connection header received
0619 09:01:30.685121 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:30.685122 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:30.685131 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:30.685208 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 09:01:30.685410 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:30.685918 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:30.687385 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:30.687742 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:30.687803 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:30.687814 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0619 09:01:30.687980 (+ 166us) server_negotiation.cc:299] Negotiation successful
0619 09:01:30.688022 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260619 09:01:30.688167 7857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:30.684875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34875 (local address 127.0.98.1:48313)
0619 09:01:30.685239 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0619 09:01:30.685247 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0619 09:01:30.685290 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:01:30.685419 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:01:30.685420 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:01:30.685421 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 09:01:30.685495 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:30.685496 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:30.686046 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:30.686046 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:30.687310 (+ 1264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:30.687314 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:30.687880 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:30.687882 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:30.687932 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:30.687935 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 09:01:30.687988 (+ 53us) client_negotiation.cc:240] Negotiation successful
0619 09:01:30.688034 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":342,"mutex_wait_us":229,"thread_start_us":122,"threads_started":1}
I20260619 09:01:30.688499 7572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44c9c858d29a4610803799759d801474" candidate_uuid: "551feb9174794395a270058ce5ed45c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00cca0af6ec34749baea4d021c20755e" is_pre_election: true
I20260619 09:01:30.688599 7572 raft_consensus.cc:2454] T 44c9c858d29a4610803799759d801474 P 00cca0af6ec34749baea4d021c20755e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 551feb9174794395a270058ce5ed45c5 in term 0.
I20260619 09:01:30.688598 7708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44c9c858d29a4610803799759d801474" candidate_uuid: "551feb9174794395a270058ce5ed45c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5cc45b850424e7da92acf9089450d80" is_pre_election: true
I20260619 09:01:30.688714 7708 raft_consensus.cc:2454] T 44c9c858d29a4610803799759d801474 P a5cc45b850424e7da92acf9089450d80 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 551feb9174794395a270058ce5ed45c5 in term 0.
I20260619 09:01:30.688791 7369 leader_election.cc:302] T 44c9c858d29a4610803799759d801474 P 551feb9174794395a270058ce5ed45c5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00cca0af6ec34749baea4d021c20755e, 551feb9174794395a270058ce5ed45c5; no voters:
I20260619 09:01:30.688913 7848 raft_consensus.cc:2781] T 44c9c858d29a4610803799759d801474 P 551feb9174794395a270058ce5ed45c5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 09:01:30.688958 7848 raft_consensus.cc:482] T 44c9c858d29a4610803799759d801474 P 551feb9174794395a270058ce5ed45c5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 09:01:30.688978 7848 raft_consensus.cc:3037] T 44c9c858d29a4610803799759d801474 P 551feb9174794395a270058ce5ed45c5 [term 0 FOLLOWER]: Advancing to term 1
I20260619 09:01:30.689656 7848 raft_consensus.cc:504] T 44c9c858d29a4610803799759d801474 P 551feb9174794395a270058ce5ed45c5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "551feb9174794395a270058ce5ed45c5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40909 } } peers { permanent_uuid: "a5cc45b850424e7da92acf9089450d80" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34875 } } peers { permanent_uuid: "00cca0af6ec34749baea4d021c20755e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37741 } }
I20260619 09:01:30.689778 7848 leader_election.cc:288] T 44c9c858d29a4610803799759d801474 P 551feb9174794395a270058ce5ed45c5 [CANDIDATE]: Term 1 election: Requested vote from peers a5cc45b850424e7da92acf9089450d80 (127.0.98.3:34875), 00cca0af6ec34749baea4d021c20755e (127.0.98.2:37741)
I20260619 09:01:30.689913 7708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44c9c858d29a4610803799759d801474" candidate_uuid: "551feb9174794395a270058ce5ed45c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5cc45b850424e7da92acf9089450d80"
I20260619 09:01:30.689924 7572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44c9c858d29a4610803799759d801474" candidate_uuid: "551feb9174794395a270058ce5ed45c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00cca0af6ec34749baea4d021c20755e"
I20260619 09:01:30.689980 7572 raft_consensus.cc:3037] T 44c9c858d29a4610803799759d801474 P 00cca0af6ec34749baea4d021c20755e [term 0 FOLLOWER]: Advancing to term 1
I20260619 09:01:30.689985 7708 raft_consensus.cc:3037] T 44c9c858d29a4610803799759d801474 P a5cc45b850424e7da92acf9089450d80 [term 0 FOLLOWER]: Advancing to term 1
I20260619 09:01:30.690526 7572 raft_consensus.cc:2454] T 44c9c858d29a4610803799759d801474 P 00cca0af6ec34749baea4d021c20755e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 551feb9174794395a270058ce5ed45c5 in term 1.
I20260619 09:01:30.690666 7708 raft_consensus.cc:2454] T 44c9c858d29a4610803799759d801474 P a5cc45b850424e7da92acf9089450d80 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 551feb9174794395a270058ce5ed45c5 in term 1.
I20260619 09:01:30.690676 7369 leader_election.cc:302] T 44c9c858d29a4610803799759d801474 P 551feb9174794395a270058ce5ed45c5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00cca0af6ec34749baea4d021c20755e, 551feb9174794395a270058ce5ed45c5; no voters:
I20260619 09:01:30.690761 7848 raft_consensus.cc:2781] T 44c9c858d29a4610803799759d801474 P 551feb9174794395a270058ce5ed45c5 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 09:01:30.690924 7848 raft_consensus.cc:686] T 44c9c858d29a4610803799759d801474 P 551feb9174794395a270058ce5ed45c5 [term 1 LEADER]: Becoming Leader. State: Replica: 551feb9174794395a270058ce5ed45c5, State: Running, Role: LEADER
I20260619 09:01:30.690999 7848 consensus_queue.cc:237] T 44c9c858d29a4610803799759d801474 P 551feb9174794395a270058ce5ed45c5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "551feb9174794395a270058ce5ed45c5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40909 } } peers { permanent_uuid: "a5cc45b850424e7da92acf9089450d80" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34875 } } peers { permanent_uuid: "00cca0af6ec34749baea4d021c20755e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37741 } }
I20260619 09:01:30.691202 7857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:30.691104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44403 (local address 127.0.98.1:55511)
0619 09:01:30.691141 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0619 09:01:30.691175 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
I20260619 09:01:30.691721 7114 catalog_manager.cc:5495] T 44c9c858d29a4610803799759d801474 P 551feb9174794395a270058ce5ed45c5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 551feb9174794395a270058ce5ed45c5 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "551feb9174794395a270058ce5ed45c5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "551feb9174794395a270058ce5ed45c5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40909 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a5cc45b850424e7da92acf9089450d80" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34875 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "00cca0af6ec34749baea4d021c20755e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37741 } health_report { overall_health: UNKNOWN } } }
I20260619 09:01:30.724285 7840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:30.722043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51180 (local address 127.0.98.1:40909)
0619 09:01:30.722101 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:30.722103 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:30.722112 (+ 9us) server_negotiation.cc:407] Connection header received
0619 09:01:30.722170 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:30.722170 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:30.722191 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:30.722262 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 09:01:30.722441 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:30.722938 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:30.723623 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:30.723802 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:30.723847 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:30.724015 (+ 168us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 09:01:30.724118 (+ 103us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 09:01:30.724167 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0619 09:01:30.724195 (+ 28us) server_negotiation.cc:299] Negotiation successful
0619 09:01:30.724212 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260619 09:01:30.725421 7572 raft_consensus.cc:1261] T 44c9c858d29a4610803799759d801474 P 00cca0af6ec34749baea4d021c20755e [term 1 FOLLOWER]: Refusing update from remote peer 551feb9174794395a270058ce5ed45c5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 09:01:30.725461 7708 raft_consensus.cc:1261] T 44c9c858d29a4610803799759d801474 P a5cc45b850424e7da92acf9089450d80 [term 1 FOLLOWER]: Refusing update from remote peer 551feb9174794395a270058ce5ed45c5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 09:01:30.725680 7848 consensus_queue.cc:1035] T 44c9c858d29a4610803799759d801474 P 551feb9174794395a270058ce5ed45c5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00cca0af6ec34749baea4d021c20755e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37741 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 09:01:30.725803 7858 consensus_queue.cc:1035] T 44c9c858d29a4610803799759d801474 P 551feb9174794395a270058ce5ed45c5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5cc45b850424e7da92acf9089450d80" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 09:01:30.726493 7856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:30.726390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44403 (local address 127.0.98.3:51431)
0619 09:01:30.726432 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0619 09:01:30.726462 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260619 09:01:30.726720 7855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:30.726599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44403 (local address 127.0.98.2:37439)
0619 09:01:30.726650 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0619 09:01:30.726680 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260619 09:01:30.727419 7861 mvcc.cc:205] Tried to move back new op lower bound from 7298497293208379392 to 7298497293069991936. Current Snapshot: MvccSnapshot[applied={T|T < 7298497293208379392}]
I20260619 09:01:30.727941 7857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:30.727785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44403 (local address 127.0.98.1:35927)
0619 09:01:30.727886 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0619 09:01:30.727913 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260619 09:01:30.728097 7856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:30.727990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44403 (local address 127.0.98.3:55413)
0619 09:01:30.728038 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0619 09:01:30.728059 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260619 09:01:30.729761 7855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:30.729675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44403 (local address 127.0.98.2:42809)
0619 09:01:30.729712 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0619 09:01:30.729732 (+ 20us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
I20260619 09:01:30.730057 7115 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43284:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 09:01:30.730140 7115 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43284:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 09:01:30.731412 7115 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 13ea0fb4e8094bd8bda6763a0a07c2c5: Sending DeleteTablet for 3 replicas of tablet 44c9c858d29a4610803799759d801474
I20260619 09:01:30.731695 7552 tablet_service.cc:1514] Processing DeleteTablet for tablet 44c9c858d29a4610803799759d801474 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 09:01:30 UTC) from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:43744
I20260619 09:01:30.731719 7687 tablet_service.cc:1514] Processing DeleteTablet for tablet 44c9c858d29a4610803799759d801474 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 09:01:30 UTC) from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:38106
I20260619 09:01:30.731776 7416 tablet_service.cc:1514] Processing DeleteTablet for tablet 44c9c858d29a4610803799759d801474 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 09:01:30 UTC) from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:51176
I20260619 09:01:30.731895 7871 tablet_replica.cc:330] T 44c9c858d29a4610803799759d801474 P 00cca0af6ec34749baea4d021c20755e: stopping tablet replica
I20260619 09:01:30.731952 7871 raft_consensus.cc:2229] T 44c9c858d29a4610803799759d801474 P 00cca0af6ec34749baea4d021c20755e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 09:01:30.731998 7871 raft_consensus.cc:2258] T 44c9c858d29a4610803799759d801474 P 00cca0af6ec34749baea4d021c20755e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 09:01:30.732093 7872 tablet_replica.cc:330] T 44c9c858d29a4610803799759d801474 P a5cc45b850424e7da92acf9089450d80: stopping tablet replica
I20260619 09:01:30.732143 7872 raft_consensus.cc:2229] T 44c9c858d29a4610803799759d801474 P a5cc45b850424e7da92acf9089450d80 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 09:01:30.732184 7872 raft_consensus.cc:2258] T 44c9c858d29a4610803799759d801474 P a5cc45b850424e7da92acf9089450d80 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 09:01:30.732177 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ez5NB/build/release/bin/kudu with pid 7353
I20260619 09:01:30.732770 7872 ts_tablet_manager.cc:1905] T 44c9c858d29a4610803799759d801474 P a5cc45b850424e7da92acf9089450d80: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 09:01:30.734138 7872 ts_tablet_manager.cc:1918] T 44c9c858d29a4610803799759d801474 P a5cc45b850424e7da92acf9089450d80: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 09:01:30.734186 7872 log.cc:1196] T 44c9c858d29a4610803799759d801474 P a5cc45b850424e7da92acf9089450d80: Deleting WAL directory at /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781859658134510-392-0/minicluster-data/ts-2/wal/wals/44c9c858d29a4610803799759d801474
I20260619 09:01:30.734390 7872 ts_tablet_manager.cc:1939] T 44c9c858d29a4610803799759d801474 P a5cc45b850424e7da92acf9089450d80: Deleting consensus metadata
I20260619 09:01:30.734666 7100 catalog_manager.cc:4841] TS a5cc45b850424e7da92acf9089450d80 (127.0.98.3:34875): tablet 44c9c858d29a4610803799759d801474 (table test-table [id=3ee55b69707e4bbcba49751c8411319f]) successfully deleted
I20260619 09:01:30.737876 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ez5NB/build/release/bin/kudu with pid 7489
I20260619 09:01:30.737886 7841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:30.737760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40909 (local address 127.0.0.1:51190)
0619 09:01:30.737806 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0619 09:01:30.737834 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260619 09:01:30.738987 7101 catalog_manager.cc:4584] TS 551feb9174794395a270058ce5ed45c5 (127.0.98.1:40909): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 44c9c858d29a4610803799759d801474: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40909: connect: Connection refused (error 111)
I20260619 09:01:30.742084 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ez5NB/build/release/bin/kudu with pid 7625
I20260619 09:01:30.742157 7841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:30.742053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37741 (local address 127.0.0.1:43758)
0619 09:01:30.742096 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0619 09:01:30.742127 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:37741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
I20260619 09:01:30.746655 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ez5NB/build/release/bin/kudu with pid 6981
I20260619 09:01:30.750448 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ez5NB/build/release/bin/kudu with pid 7169
2026-06-19T09:01:30Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (4753 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781859658134510-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 09:01:30 dist-test-slave-mtdr krb5kdc[7876](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 09:01:30 dist-test-slave-mtdr krb5kdc[7876](info): set up 2 sockets
Jun 19 09:01:30 dist-test-slave-mtdr krb5kdc[7876](info): commencing operation
krb5kdc: starting...
W20260619 09:01:31.702991 392 mini_kdc.cc:121] Time spent starting KDC: real 0.932s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781859658134510-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781859658134510-392-0/krb5kdc/test-user.keytab.
Jun 19 09:01:31 dist-test-slave-mtdr krb5kdc[7876](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859691, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T09:01:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T09:01:31Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781859658134510-392-0/krb5kdc/oryx_127.0.98.62.keytab.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781859658134510-392-0/krb5kdc/oryx_127.0.98.62.keytab.
I20260619 09:01:31.811607 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ez5NB/build/release/bin/kudu
/tmp/dist-test-task4ez5NB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781859658134510-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781859658134510-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781859658134510-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781859658134510-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35301
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781859658134510-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35301
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781859658134510-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781859658134510-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781859658134510-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 09:01:31.895980 7889 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 09:01:31.896162 7889 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 09:01:31.896190 7889 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 09:01:31.897296 7889 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 09:01:31.897337 7889 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260619 09:01:31.897349 7889 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 09:01:31.897360 7889 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 09:01:31.897373 7889 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 09:01:31.897395 7889 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 09:01:31.898782 7889 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781859658134510-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781859658134510-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35301
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781859658134510-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35301
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781859658134510-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7889
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781859658134510-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 09:01:31.899022 7889 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 09:01:31.899225 7889 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 09:01:31.903580 7895 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:01:31.903656 7894 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:01:31.903587 7897 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:01:31.903707 7889 server_base.cc:1034] running on GCE node
I20260619 09:01:31.903963 7889 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 09:01:31.904191 7889 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 09:01:31.905329 7889 hybrid_clock.cc:648] HybridClock initialized: now 1781859691905315 us; error 30 us; skew 500 ppm
Jun 19 09:01:31 dist-test-slave-mtdr krb5kdc[7876](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859691, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 09:01:31.907635 7889 init.cc:377] Logged in from keytab as oryx/127.0.98.62@KRBTEST.COM (short username oryx)
I20260619 09:01:31.908044 7889 webserver.cc:460] Webserver started at http://127.0.98.62:40073/ using document root <none> and password file <none>
I20260619 09:01:31.908238 7889 fs_manager.cc:359] Metadata directory not provided
I20260619 09:01:31.908280 7889 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 09:01:31.908396 7889 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 09:01:31.909337 7889 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781859658134510-392-0/minicluster-data/master-0/data/instance:
uuid: "75e0d761165f4e3f9aa72ac34214c70c"
format_stamp: "Formatted at 2026-06-19 09:01:31 on dist-test-slave-mtdr"
server_key: "0a060caf1de16f0d1a019592a26244eb"
server_key_iv: "c2575a972894530680006e50df6ca2ba"
server_key_version: "encryptionkey@0"
I20260619 09:01:31.909641 7889 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781859658134510-392-0/minicluster-data/master-0/wal/instance:
uuid: "75e0d761165f4e3f9aa72ac34214c70c"
format_stamp: "Formatted at 2026-06-19 09:01:31 on dist-test-slave-mtdr"
server_key: "0a060caf1de16f0d1a019592a26244eb"
server_key_iv: "c2575a972894530680006e50df6ca2ba"
server_key_version: "encryptionkey@0"
I20260619 09:01:31.910812 7889 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260619 09:01:31.911517 7904 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 09:01:31.911659 7889 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 09:01:31.911731 7889 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781859658134510-392-0/minicluster-data/master-0/data,/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781859658134510-392-0/minicluster-data/master-0/wal
uuid: "75e0d761165f4e3f9aa72ac34214c70c"
format_stamp: "Formatted at 2026-06-19 09:01:31 on dist-test-slave-mtdr"
server_key: "0a060caf1de16f0d1a019592a26244eb"
server_key_iv: "c2575a972894530680006e50df6ca2ba"
server_key_version: "encryptionkey@0"
I20260619 09:01:31.911792 7889 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781859658134510-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781859658134510-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781859658134510-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 09:01:31.925230 7889 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 09:01:31.926910 7889 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 09:01:31.927032 7889 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 09:01:31.931238 7889 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35301
I20260619 09:01:31.931286 7966 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35301 every 8 connection(s)
I20260619 09:01:31.931593 7889 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781859658134510-392-0/minicluster-data/master-0/data/info.pb
I20260619 09:01:31.932145 7967 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 09:01:31.934273 7967 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 75e0d761165f4e3f9aa72ac34214c70c: Bootstrap starting.
I20260619 09:01:31.934787 7967 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 75e0d761165f4e3f9aa72ac34214c70c: Neither blocks nor log segments found. Creating new log.
I20260619 09:01:31.935104 7967 log.cc:824] T 00000000000000000000000000000000 P 75e0d761165f4e3f9aa72ac34214c70c: Log is configured to *not* fsync() on all Append() calls
I20260619 09:01:31.935681 7967 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 75e0d761165f4e3f9aa72ac34214c70c: No bootstrap required, opened a new log
I20260619 09:01:31.936300 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ez5NB/build/release/bin/kudu as pid 7889
I20260619 09:01:31.936376 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781859658134510-392-0/minicluster-data/master-0/wal/instance
I20260619 09:01:31.936527 392 external_mini_cluster.cc:1428] Setting key 202c268537cb4527302bbfb888486ec1
I20260619 09:01:31.937994 7967 raft_consensus.cc:348] T 00000000000000000000000000000000 P 75e0d761165f4e3f9aa72ac34214c70c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75e0d761165f4e3f9aa72ac34214c70c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35301 } }
I20260619 09:01:31.938158 7967 raft_consensus.cc:374] T 00000000000000000000000000000000 P 75e0d761165f4e3f9aa72ac34214c70c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 09:01:31.938190 7967 raft_consensus.cc:729] T 00000000000000000000000000000000 P 75e0d761165f4e3f9aa72ac34214c70c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75e0d761165f4e3f9aa72ac34214c70c, State: Initialized, Role: FOLLOWER
I20260619 09:01:31.938310 7967 consensus_queue.cc:260] T 00000000000000000000000000000000 P 75e0d761165f4e3f9aa72ac34214c70c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75e0d761165f4e3f9aa72ac34214c70c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35301 } }
I20260619 09:01:31.938382 7967 raft_consensus.cc:388] T 00000000000000000000000000000000 P 75e0d761165f4e3f9aa72ac34214c70c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 09:01:31.938405 7967 raft_consensus.cc:482] T 00000000000000000000000000000000 P 75e0d761165f4e3f9aa72ac34214c70c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 09:01:31.938452 7967 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 75e0d761165f4e3f9aa72ac34214c70c [term 0 FOLLOWER]: Advancing to term 1
I20260619 09:01:31.939028 7967 raft_consensus.cc:504] T 00000000000000000000000000000000 P 75e0d761165f4e3f9aa72ac34214c70c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75e0d761165f4e3f9aa72ac34214c70c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35301 } }
I20260619 09:01:31.939138 7967 leader_election.cc:302] T 00000000000000000000000000000000 P 75e0d761165f4e3f9aa72ac34214c70c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 75e0d761165f4e3f9aa72ac34214c70c; no voters:
I20260619 09:01:31.939277 7967 leader_election.cc:288] T 00000000000000000000000000000000 P 75e0d761165f4e3f9aa72ac34214c70c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 09:01:31.939352 7972 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 75e0d761165f4e3f9aa72ac34214c70c [term 1 FOLLOWER]: Leader election won for term 1
I20260619 09:01:31.939494 7972 raft_consensus.cc:686] T 00000000000000000000000000000000 P 75e0d761165f4e3f9aa72ac34214c70c [term 1 LEADER]: Becoming Leader. State: Replica: 75e0d761165f4e3f9aa72ac34214c70c, State: Running, Role: LEADER
I20260619 09:01:31.939503 7967 sys_catalog.cc:564] T 00000000000000000000000000000000 P 75e0d761165f4e3f9aa72ac34214c70c [sys.catalog]: configured and running, proceeding with master startup.
I20260619 09:01:31.939608 7972 consensus_queue.cc:237] T 00000000000000000000000000000000 P 75e0d761165f4e3f9aa72ac34214c70c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75e0d761165f4e3f9aa72ac34214c70c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35301 } }
I20260619 09:01:31.940033 7972 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75e0d761165f4e3f9aa72ac34214c70c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 75e0d761165f4e3f9aa72ac34214c70c. Latest consensus state: current_term: 1 leader_uuid: "75e0d761165f4e3f9aa72ac34214c70c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75e0d761165f4e3f9aa72ac34214c70c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35301 } } }
I20260619 09:01:31.940105 7972 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75e0d761165f4e3f9aa72ac34214c70c [sys.catalog]: This master's current role is: LEADER
I20260619 09:01:31.940238 7973 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75e0d761165f4e3f9aa72ac34214c70c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "75e0d761165f4e3f9aa72ac34214c70c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75e0d761165f4e3f9aa72ac34214c70c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35301 } } }
I20260619 09:01:31.940344 7973 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75e0d761165f4e3f9aa72ac34214c70c [sys.catalog]: This master's current role is: LEADER
I20260619 09:01:31.940451 7981 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 09:01:31.940896 7981 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 09:01:31.942909 7981 catalog_manager.cc:1347] Generated new cluster ID: 7aedc9302a5c40bea87b9ef2340da6df
I20260619 09:01:31.943002 7981 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 19 09:01:31 dist-test-slave-mtdr krb5kdc[7876](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859691, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260619 09:01:31.948364 7971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:31.936947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33924 (local address 127.0.98.62:35301)
0619 09:01:31.937159 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:31.937169 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:31.937202 (+ 33us) server_negotiation.cc:407] Connection header received
0619 09:01:31.937962 (+ 760us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:31.937971 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:31.938034 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:31.938172 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:01:31.938846 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:31.940198 (+ 1352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:31.941231 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:31.941777 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:31.941880 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:31.944524 (+ 2644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:01:31.944529 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:01:31.944529 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:01:31.944541 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:01:31.946849 (+ 2308us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:01:31.947230 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:01:31.947232 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:01:31.947233 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:01:31.947341 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:01:31.947524 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:01:31.947524 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:01:31.947525 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:01:31.947799 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:01:31.947928 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0619 09:01:31.948034 (+ 106us) server_negotiation.cc:299] Negotiation successful
0619 09:01:31.948155 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":98,"threads_started":1}
I20260619 09:01:31.949889 7981 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 09:01:31.950344 7981 catalog_manager.cc:1504] Loading token signing keys...
I20260619 09:01:31.953413 7981 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 75e0d761165f4e3f9aa72ac34214c70c: Generated new TSK 0
I20260619 09:01:31.953550 7981 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781859658134510-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781859658134510-392-0/krb5kdc/oryx_127.0.98.1.keytab.
I20260619 09:01:31.977614 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ez5NB/build/release/bin/kudu
/tmp/dist-test-task4ez5NB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781859658134510-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781859658134510-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781859658134510-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781859658134510-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35301
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781859658134510-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781859658134510-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781859658134510-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781859658134510-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 09:01:32.054250 7993 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 09:01:32.054437 7993 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 09:01:32.054468 7993 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 09:01:32.055647 7993 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 09:01:32.055686 7993 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 09:01:32.055713 7993 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 09:01:32.055737 7993 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 09:01:32.057191 7993 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781859658134510-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781859658134510-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781859658134510-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781859658134510-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:35301
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7993
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781859658134510-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 09:01:32.057413 7993 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 09:01:32.057608 7993 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 09:01:32.063076 7999 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:01:32.063185 7998 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:01:32.063109 8001 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:01:32.063262 7993 server_base.cc:1034] running on GCE node
I20260619 09:01:32.063509 7993 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 09:01:32.063726 7993 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 09:01:32.064857 7993 hybrid_clock.cc:648] HybridClock initialized: now 1781859692064848 us; error 34 us; skew 500 ppm
Jun 19 09:01:32 dist-test-slave-mtdr krb5kdc[7876](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859692, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 09:01:32.067122 7993 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260619 09:01:32.067509 7993 webserver.cc:460] Webserver started at http://127.0.98.1:45091/ using document root <none> and password file <none>
I20260619 09:01:32.067677 7993 fs_manager.cc:359] Metadata directory not provided
I20260619 09:01:32.067713 7993 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 09:01:32.067799 7993 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 09:01:32.068744 7993 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781859658134510-392-0/minicluster-data/ts-0/data/instance:
uuid: "c9d6290881cd499dbaf9dd32d03583aa"
format_stamp: "Formatted at 2026-06-19 09:01:32 on dist-test-slave-mtdr"
server_key: "1cbe31fb8a8de0bb709ce936d840c5e1"
server_key_iv: "85fa957095717b19632acc41dc731032"
server_key_version: "encryptionkey@0"
I20260619 09:01:32.069031 7993 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781859658134510-392-0/minicluster-data/ts-0/wal/instance:
uuid: "c9d6290881cd499dbaf9dd32d03583aa"
format_stamp: "Formatted at 2026-06-19 09:01:32 on dist-test-slave-mtdr"
server_key: "1cbe31fb8a8de0bb709ce936d840c5e1"
server_key_iv: "85fa957095717b19632acc41dc731032"
server_key_version: "encryptionkey@0"
I20260619 09:01:32.070145 7993 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260619 09:01:32.070741 8008 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 09:01:32.070959 7993 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260619 09:01:32.071033 7993 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781859658134510-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781859658134510-392-0/minicluster-data/ts-0/wal
uuid: "c9d6290881cd499dbaf9dd32d03583aa"
format_stamp: "Formatted at 2026-06-19 09:01:32 on dist-test-slave-mtdr"
server_key: "1cbe31fb8a8de0bb709ce936d840c5e1"
server_key_iv: "85fa957095717b19632acc41dc731032"
server_key_version: "encryptionkey@0"
I20260619 09:01:32.071097 7993 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781859658134510-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781859658134510-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781859658134510-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 09:01:32.084489 7993 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 09:01:32.086124 7993 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 09:01:32.086242 7993 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 09:01:32.086710 7993 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 09:01:32.086750 7993 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 09:01:32.086769 7993 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 09:01:32.086781 7993 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 09:01:32.093298 7993 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44937
I20260619 09:01:32.093454 8123 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44937 every 8 connection(s)
Jun 19 09:01:32 dist-test-slave-mtdr krb5kdc[7876](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859692, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260619 09:01:32.093695 7993 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781859658134510-392-0/minicluster-data/ts-0/data/info.pb
I20260619 09:01:32.096290 7971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:32.087173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43037 (local address 127.0.98.62:35301)
0619 09:01:32.087243 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:32.087245 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:32.087421 (+ 176us) server_negotiation.cc:407] Connection header received
0619 09:01:32.088079 (+ 658us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:32.088080 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:32.088090 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:32.088132 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:01:32.088950 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:32.089742 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:32.091202 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:32.091517 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:32.091591 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:32.094016 (+ 2425us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:01:32.094018 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:01:32.094018 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:01:32.094029 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:01:32.095247 (+ 1218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:01:32.095593 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:01:32.095594 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:01:32.095594 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:01:32.095636 (+ 42us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:01:32.095813 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:01:32.095814 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:01:32.095814 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:01:32.095906 (+ 92us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:01:32.095950 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0619 09:01:32.096155 (+ 205us) server_negotiation.cc:299] Negotiation successful
0619 09:01:32.096223 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260619 09:01:32.096426 8017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:32.087135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35301 (local address 127.0.98.1:43037)
0619 09:01:32.087286 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0619 09:01:32.087331 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 09:01:32.087980 (+ 649us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:01:32.088216 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:01:32.088221 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:01:32.088560 (+ 339us) client_negotiation.cc:189] Negotiated authn=SASL
0619 09:01:32.088837 (+ 277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:32.088844 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:32.089829 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:32.089830 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:32.091119 (+ 1289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:32.091122 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:32.091597 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:32.091598 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:32.091740 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:32.092280 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 09:01:32.092290 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 09:01:32.093931 (+ 1641us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 09:01:32.095322 (+ 1391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:01:32.095327 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:01:32.095329 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:01:32.095537 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:01:32.095681 (+ 144us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:01:32.095681 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:01:32.095681 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:01:32.095771 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:01:32.095949 (+ 178us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 09:01:32.095952 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 09:01:32.096061 (+ 109us) client_negotiation.cc:769] Sending connection context
0619 09:01:32.096137 (+ 76us) client_negotiation.cc:240] Negotiation successful
0619 09:01:32.096250 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":73,"threads_started":1}
I20260619 09:01:32.096884 8124 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35301
I20260619 09:01:32.096967 8124 heartbeater.cc:464] Registering TS with master...
I20260619 09:01:32.097131 8124 heartbeater.cc:511] Master 127.0.98.62:35301 requested a full tablet report, sending...
I20260619 09:01:32.097668 7920 ts_manager.cc:194] Registered new tserver with Master: c9d6290881cd499dbaf9dd32d03583aa (127.0.98.1:44937)
I20260619 09:01:32.098338 7920 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:43037
I20260619 09:01:32.101882 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ez5NB/build/release/bin/kudu as pid 7993
I20260619 09:01:32.101963 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781859658134510-392-0/minicluster-data/ts-0/wal/instance
I20260619 09:01:32.102141 392 external_mini_cluster.cc:1428] Setting key 36941bd1a0a7ca915ab6c31cf26aefcb
I20260619 09:01:32.102541 8130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:32.097782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35301 (local address 127.0.98.1:51569)
0619 09:01:32.097909 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0619 09:01:32.097918 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 09:01:32.097970 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:01:32.098074 (+ 104us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:01:32.098075 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:01:32.098231 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0619 09:01:32.098277 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:32.098278 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:32.099147 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:32.099148 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:32.099643 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:32.099644 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:32.099932 (+ 288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:32.099933 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:32.099976 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:32.100349 (+ 373us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 09:01:32.100353 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 09:01:32.100603 (+ 250us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 09:01:32.101687 (+ 1084us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:01:32.101688 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:01:32.101688 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:01:32.101838 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:01:32.102008 (+ 170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:01:32.102009 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:01:32.102009 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:01:32.102063 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:01:32.102271 (+ 208us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 09:01:32.102272 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 09:01:32.102311 (+ 39us) client_negotiation.cc:769] Sending connection context
0619 09:01:32.102377 (+ 66us) client_negotiation.cc:240] Negotiation successful
0619 09:01:32.102466 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":60,"threads_started":1}
I20260619 09:01:32.102599 7971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:32.097838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51569 (local address 127.0.98.62:35301)
0619 09:01:32.097877 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:32.097879 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:32.097970 (+ 91us) server_negotiation.cc:407] Connection header received
0619 09:01:32.098020 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:32.098020 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:32.098029 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:32.098072 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:01:32.098338 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:32.099068 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:32.099705 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:32.099881 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:32.099927 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:32.100675 (+ 748us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:01:32.100676 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:01:32.100677 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:01:32.100682 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:01:32.101618 (+ 936us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:01:32.101894 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:01:32.101895 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:01:32.101895 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:01:32.101944 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:01:32.102112 (+ 168us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:01:32.102113 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:01:32.102113 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:01:32.102222 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:01:32.102279 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0619 09:01:32.102429 (+ 150us) server_negotiation.cc:299] Negotiation successful
0619 09:01:32.102510 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781859658134510-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781859658134510-392-0/krb5kdc/oryx_127.0.98.2.keytab.
I20260619 09:01:32.124843 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ez5NB/build/release/bin/kudu
/tmp/dist-test-task4ez5NB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781859658134510-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781859658134510-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781859658134510-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781859658134510-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35301
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781859658134510-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781859658134510-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781859658134510-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781859658134510-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 09:01:32.204149 8133 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 09:01:32.204356 8133 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 09:01:32.204388 8133 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 09:01:32.205521 8133 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 09:01:32.205560 8133 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 09:01:32.205587 8133 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 09:01:32.205610 8133 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 09:01:32.207083 8133 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781859658134510-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781859658134510-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781859658134510-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781859658134510-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:35301
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8133
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781859658134510-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 09:01:32.207314 8133 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 09:01:32.207506 8133 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 09:01:32.212170 8138 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:01:32.212181 8141 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:01:32.212167 8139 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:01:32.212374 8133 server_base.cc:1034] running on GCE node
I20260619 09:01:32.212574 8133 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 09:01:32.212762 8133 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 09:01:32.213850 8133 hybrid_clock.cc:648] HybridClock initialized: now 1781859692213833 us; error 29 us; skew 500 ppm
Jun 19 09:01:32 dist-test-slave-mtdr krb5kdc[7876](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859692, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 09:01:32.216188 8133 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260619 09:01:32.216599 8133 webserver.cc:460] Webserver started at http://127.0.98.2:42065/ using document root <none> and password file <none>
I20260619 09:01:32.216804 8133 fs_manager.cc:359] Metadata directory not provided
I20260619 09:01:32.216850 8133 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 09:01:32.216974 8133 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 09:01:32.217965 8133 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781859658134510-392-0/minicluster-data/ts-1/data/instance:
uuid: "f06926dc12064dd78ea8de2e1a4259ed"
format_stamp: "Formatted at 2026-06-19 09:01:32 on dist-test-slave-mtdr"
server_key: "a62c41b4a0821b9dbef49f086f56d85e"
server_key_iv: "329a49789d1a2a4ce7b9bf159484873f"
server_key_version: "encryptionkey@0"
I20260619 09:01:32.218290 8133 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781859658134510-392-0/minicluster-data/ts-1/wal/instance:
uuid: "f06926dc12064dd78ea8de2e1a4259ed"
format_stamp: "Formatted at 2026-06-19 09:01:32 on dist-test-slave-mtdr"
server_key: "a62c41b4a0821b9dbef49f086f56d85e"
server_key_iv: "329a49789d1a2a4ce7b9bf159484873f"
server_key_version: "encryptionkey@0"
I20260619 09:01:32.219584 8133 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 09:01:32.220341 8148 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 09:01:32.220503 8133 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260619 09:01:32.220575 8133 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781859658134510-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781859658134510-392-0/minicluster-data/ts-1/wal
uuid: "f06926dc12064dd78ea8de2e1a4259ed"
format_stamp: "Formatted at 2026-06-19 09:01:32 on dist-test-slave-mtdr"
server_key: "a62c41b4a0821b9dbef49f086f56d85e"
server_key_iv: "329a49789d1a2a4ce7b9bf159484873f"
server_key_version: "encryptionkey@0"
I20260619 09:01:32.220638 8133 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781859658134510-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781859658134510-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781859658134510-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 09:01:32.231585 8133 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 09:01:32.233469 8133 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 09:01:32.233599 8133 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 09:01:32.234052 8133 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 09:01:32.234093 8133 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 09:01:32.234122 8133 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 09:01:32.234146 8133 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 09:01:32 dist-test-slave-mtdr krb5kdc[7876](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859692, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260619 09:01:32.240795 8263 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41951 every 8 connection(s)
I20260619 09:01:32.240698 8133 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41951
I20260619 09:01:32.241268 8133 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781859658134510-392-0/minicluster-data/ts-1/data/info.pb
I20260619 09:01:32.243516 7971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:32.234477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56413 (local address 127.0.98.62:35301)
0619 09:01:32.234524 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:32.234526 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:32.234708 (+ 182us) server_negotiation.cc:407] Connection header received
0619 09:01:32.235260 (+ 552us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:32.235261 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:32.235273 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:32.235316 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:01:32.235942 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:32.236685 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:32.238184 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:32.238483 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:32.238552 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:32.240960 (+ 2408us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:01:32.240962 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:01:32.240962 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:01:32.240969 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:01:32.242241 (+ 1272us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:01:32.242598 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:01:32.242599 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:01:32.242600 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:01:32.242655 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:01:32.242928 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:01:32.242929 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:01:32.242929 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:01:32.243046 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:01:32.243101 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0619 09:01:32.243355 (+ 254us) server_negotiation.cc:299] Negotiation successful
0619 09:01:32.243429 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260619 09:01:32.243664 8157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:32.234474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35301 (local address 127.0.98.2:56413)
0619 09:01:32.234601 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0619 09:01:32.234633 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0619 09:01:32.235169 (+ 536us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:01:32.235388 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:01:32.235392 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:01:32.235683 (+ 291us) client_negotiation.cc:189] Negotiated authn=SASL
0619 09:01:32.235859 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:32.235863 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:32.236799 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:32.236800 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:32.238097 (+ 1297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:32.238100 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:32.238580 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:32.238581 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:32.238726 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:32.239175 (+ 449us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 09:01:32.239186 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 09:01:32.240877 (+ 1691us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 09:01:32.242318 (+ 1441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:01:32.242323 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:01:32.242325 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:01:32.242529 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:01:32.242706 (+ 177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:01:32.242707 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:01:32.242707 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:01:32.242804 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:01:32.243100 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 09:01:32.243107 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 09:01:32.243234 (+ 127us) client_negotiation.cc:769] Sending connection context
0619 09:01:32.243309 (+ 75us) client_negotiation.cc:240] Negotiation successful
0619 09:01:32.243453 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":69,"threads_started":1}
I20260619 09:01:32.244061 8264 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35301
I20260619 09:01:32.244130 8264 heartbeater.cc:464] Registering TS with master...
I20260619 09:01:32.244290 8264 heartbeater.cc:511] Master 127.0.98.62:35301 requested a full tablet report, sending...
I20260619 09:01:32.244613 7921 ts_manager.cc:194] Registered new tserver with Master: f06926dc12064dd78ea8de2e1a4259ed (127.0.98.2:41951)
I20260619 09:01:32.245071 7921 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:56413
I20260619 09:01:32.249032 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ez5NB/build/release/bin/kudu as pid 8133
I20260619 09:01:32.249135 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781859658134510-392-0/minicluster-data/ts-1/wal/instance
I20260619 09:01:32.249321 392 external_mini_cluster.cc:1428] Setting key 8c066b9e8aa831b794deb522457cf274
I20260619 09:01:32.249744 7971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:32.245011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34401 (local address 127.0.98.62:35301)
0619 09:01:32.245243 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:32.245244 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:32.245251 (+ 7us) server_negotiation.cc:407] Connection header received
0619 09:01:32.245266 (+ 15us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:32.245266 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:32.245273 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:32.245312 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:01:32.245607 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:32.246060 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:32.246711 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:32.246951 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:32.247011 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:32.247743 (+ 732us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:01:32.247744 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:01:32.247745 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:01:32.247751 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:01:32.248739 (+ 988us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:01:32.249019 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:01:32.249020 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:01:32.249021 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:01:32.249070 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:01:32.249219 (+ 149us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:01:32.249220 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:01:32.249220 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:01:32.249354 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:01:32.249411 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0619 09:01:32.249624 (+ 213us) server_negotiation.cc:299] Negotiation successful
0619 09:01:32.249682 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":216}
I20260619 09:01:32.249745 8270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:32.244955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35301 (local address 127.0.98.2:34401)
0619 09:01:32.245070 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0619 09:01:32.245080 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 09:01:32.245123 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:01:32.245328 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:01:32.245330 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:01:32.245496 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0619 09:01:32.245542 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:32.245544 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:32.246135 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:32.246136 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:32.246647 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:32.246648 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:32.247009 (+ 361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:32.247009 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:32.247070 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:32.247431 (+ 361us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 09:01:32.247434 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 09:01:32.247672 (+ 238us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 09:01:32.248805 (+ 1133us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:01:32.248806 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:01:32.248807 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:01:32.248955 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:01:32.249118 (+ 163us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:01:32.249119 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:01:32.249120 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:01:32.249175 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:01:32.249433 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 09:01:32.249434 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 09:01:32.249475 (+ 41us) client_negotiation.cc:769] Sending connection context
0619 09:01:32.249551 (+ 76us) client_negotiation.cc:240] Negotiation successful
0619 09:01:32.249640 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":90,"thread_start_us":63,"threads_started":1}
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781859658134510-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781859658134510-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260619 09:01:32.274891 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ez5NB/build/release/bin/kudu
/tmp/dist-test-task4ez5NB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781859658134510-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781859658134510-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781859658134510-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781859658134510-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35301
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781859658134510-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781859658134510-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781859658134510-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781859658134510-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 09:01:32.352346 8273 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 09:01:32.352550 8273 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 09:01:32.352579 8273 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 09:01:32.353744 8273 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 09:01:32.353785 8273 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 09:01:32.353812 8273 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 09:01:32.353835 8273 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 09:01:32.355322 8273 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781859658134510-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781859658134510-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781859658134510-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781859658134510-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:35301
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8273
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781859658134510-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 09:01:32.355551 8273 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 09:01:32.355748 8273 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 09:01:32.360250 8279 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:01:32.360265 8278 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:01:32.360338 8281 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:01:32.360579 8273 server_base.cc:1034] running on GCE node
I20260619 09:01:32.360765 8273 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 09:01:32.360991 8273 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 09:01:32.362161 8273 hybrid_clock.cc:648] HybridClock initialized: now 1781859692362142 us; error 35 us; skew 500 ppm
Jun 19 09:01:32 dist-test-slave-mtdr krb5kdc[7876](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859692, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 09:01:32.364477 8273 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260619 09:01:32.364916 8273 webserver.cc:460] Webserver started at http://127.0.98.3:36713/ using document root <none> and password file <none>
I20260619 09:01:32.365116 8273 fs_manager.cc:359] Metadata directory not provided
I20260619 09:01:32.365164 8273 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 09:01:32.365284 8273 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 09:01:32.366287 8273 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781859658134510-392-0/minicluster-data/ts-2/data/instance:
uuid: "0269a239f4624610a1ce99d87e3f8967"
format_stamp: "Formatted at 2026-06-19 09:01:32 on dist-test-slave-mtdr"
server_key: "2bbd062918e3bad62265ccc13a35b342"
server_key_iv: "83677ee58adcf26feefa540a6649687c"
server_key_version: "encryptionkey@0"
I20260619 09:01:32.366613 8273 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781859658134510-392-0/minicluster-data/ts-2/wal/instance:
uuid: "0269a239f4624610a1ce99d87e3f8967"
format_stamp: "Formatted at 2026-06-19 09:01:32 on dist-test-slave-mtdr"
server_key: "2bbd062918e3bad62265ccc13a35b342"
server_key_iv: "83677ee58adcf26feefa540a6649687c"
server_key_version: "encryptionkey@0"
I20260619 09:01:32.367849 8273 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 09:01:32.368515 8288 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 09:01:32.368705 8273 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 09:01:32.368806 8273 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781859658134510-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781859658134510-392-0/minicluster-data/ts-2/wal
uuid: "0269a239f4624610a1ce99d87e3f8967"
format_stamp: "Formatted at 2026-06-19 09:01:32 on dist-test-slave-mtdr"
server_key: "2bbd062918e3bad62265ccc13a35b342"
server_key_iv: "83677ee58adcf26feefa540a6649687c"
server_key_version: "encryptionkey@0"
I20260619 09:01:32.368877 8273 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781859658134510-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781859658134510-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781859658134510-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 09:01:32.381145 8273 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 09:01:32.382768 8273 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 09:01:32.382905 8273 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 09:01:32.383373 8273 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 09:01:32.383412 8273 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 09:01:32.383433 8273 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 09:01:32.383445 8273 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 09:01:32 dist-test-slave-mtdr krb5kdc[7876](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859692, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260619 09:01:32.389986 8273 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44393
I20260619 09:01:32.390051 8403 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44393 every 8 connection(s)
I20260619 09:01:32.390383 8273 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781859658134510-392-0/minicluster-data/ts-2/data/info.pb
I20260619 09:01:32.392621 7971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:32.383787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54397 (local address 127.0.98.62:35301)
0619 09:01:32.383871 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:32.383873 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:32.383998 (+ 125us) server_negotiation.cc:407] Connection header received
0619 09:01:32.384454 (+ 456us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:32.384456 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:32.384468 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:32.384551 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:01:32.385140 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:32.385880 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:32.386883 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:32.387170 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:32.387237 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:32.390148 (+ 2911us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:01:32.390150 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:01:32.390150 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:01:32.390158 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:01:32.391472 (+ 1314us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:01:32.391809 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:01:32.391809 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:01:32.391810 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:01:32.391864 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:01:32.392045 (+ 181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:01:32.392045 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:01:32.392046 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:01:32.392139 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:01:32.392184 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0619 09:01:32.392457 (+ 273us) server_negotiation.cc:299] Negotiation successful
0619 09:01:32.392558 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260619 09:01:32.392655 8297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:32.383740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35301 (local address 127.0.98.3:54397)
0619 09:01:32.383878 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0619 09:01:32.383916 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 09:01:32.384368 (+ 452us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:01:32.384627 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:01:32.384630 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:01:32.384877 (+ 247us) client_negotiation.cc:189] Negotiated authn=SASL
0619 09:01:32.385066 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:32.385071 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:32.385971 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:32.385973 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:32.386791 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:32.386794 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:32.387257 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:32.387258 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:32.387383 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:32.387862 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 09:01:32.387872 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 09:01:32.390043 (+ 2171us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 09:01:32.391536 (+ 1493us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:01:32.391543 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:01:32.391545 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:01:32.391751 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:01:32.391912 (+ 161us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:01:32.391912 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:01:32.391913 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:01:32.392003 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:01:32.392183 (+ 180us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 09:01:32.392186 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 09:01:32.392306 (+ 120us) client_negotiation.cc:769] Sending connection context
0619 09:01:32.392384 (+ 78us) client_negotiation.cc:240] Negotiation successful
0619 09:01:32.392499 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":71,"threads_started":1}
I20260619 09:01:32.393095 8404 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35301
I20260619 09:01:32.393172 8404 heartbeater.cc:464] Registering TS with master...
I20260619 09:01:32.393335 8404 heartbeater.cc:511] Master 127.0.98.62:35301 requested a full tablet report, sending...
I20260619 09:01:32.393682 7921 ts_manager.cc:194] Registered new tserver with Master: 0269a239f4624610a1ce99d87e3f8967 (127.0.98.3:44393)
I20260619 09:01:32.394069 7921 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:54397
I20260619 09:01:32.398837 7971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:32.394067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33955 (local address 127.0.98.62:35301)
0619 09:01:32.394104 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:32.394106 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:32.394168 (+ 62us) server_negotiation.cc:407] Connection header received
0619 09:01:32.394228 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:32.394229 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:32.394239 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:32.394293 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:01:32.394637 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:32.395156 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:32.395811 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:32.395982 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:32.396022 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:32.396797 (+ 775us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:01:32.396798 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:01:32.396799 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:01:32.396809 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:01:32.398051 (+ 1242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:01:32.398306 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:01:32.398307 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:01:32.398307 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:01:32.398353 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:01:32.398460 (+ 107us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:01:32.398460 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:01:32.398460 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:01:32.398567 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:01:32.398611 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0619 09:01:32.398707 (+ 96us) server_negotiation.cc:299] Negotiation successful
0619 09:01:32.398757 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260619 09:01:32.398872 8410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:32.393988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35301 (local address 127.0.98.3:33955)
0619 09:01:32.394108 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0619 09:01:32.394120 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 09:01:32.394182 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:01:32.394305 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:01:32.394306 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:01:32.394517 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0619 09:01:32.394578 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:32.394580 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:32.395237 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:32.395237 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:32.395748 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:32.395749 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:32.396044 (+ 295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:32.396045 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:32.396084 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:32.396424 (+ 340us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 09:01:32.396429 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 09:01:32.396693 (+ 264us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 09:01:32.398106 (+ 1413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:01:32.398106 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:01:32.398107 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:01:32.398249 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:01:32.398388 (+ 139us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:01:32.398388 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:01:32.398389 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:01:32.398429 (+ 40us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:01:32.398615 (+ 186us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 09:01:32.398615 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 09:01:32.398646 (+ 31us) client_negotiation.cc:769] Sending connection context
0619 09:01:32.398691 (+ 45us) client_negotiation.cc:240] Negotiation successful
0619 09:01:32.398760 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":62,"threads_started":1}
I20260619 09:01:32.399212 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ez5NB/build/release/bin/kudu as pid 8273
I20260619 09:01:32.399294 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781859658134510-392-0/minicluster-data/ts-2/wal/instance
I20260619 09:01:32.399469 392 external_mini_cluster.cc:1428] Setting key 01972c0332c990fc084fe6eb101f9968
I20260619 09:01:32.400321 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 19 09:01:32 dist-test-slave-mtdr krb5kdc[7876](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859692, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 09:01:32 dist-test-slave-mtdr krb5kdc[7876](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 09:01:32 dist-test-slave-mtdr krb5kdc[7876](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260619 09:01:32.419255 7971 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 09:01:32.414379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33926 (local address 127.0.98.62:35301)
0619 09:01:32.414424 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:32.414426 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:32.414438 (+ 12us) server_negotiation.cc:407] Connection header received
0619 09:01:32.414504 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:32.414505 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:32.414516 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:32.414577 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:01:32.415139 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:32.415639 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:32.416329 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:32.416505 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:32.416544 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:32.419178 (+ 2634us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33926: BlockingRecv error: recv got EOF from 127.0.0.1:33926 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
Jun 19 09:01:32 dist-test-slave-mtdr krb5kdc[7876](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859692, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260619 09:01:32.425719 7971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:32.419958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33932 (local address 127.0.98.62:35301)
0619 09:01:32.420010 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:32.420012 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:32.420041 (+ 29us) server_negotiation.cc:407] Connection header received
0619 09:01:32.420094 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:32.420095 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:32.420102 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:32.420144 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:01:32.420512 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:32.420973 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:32.421580 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:32.421744 (+ 164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:32.421785 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:32.423645 (+ 1860us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:01:32.423646 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:01:32.423646 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:01:32.423653 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:01:32.424929 (+ 1276us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:01:32.425193 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:01:32.425194 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:01:32.425194 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:01:32.425237 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:01:32.425350 (+ 113us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:01:32.425351 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:01:32.425351 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:01:32.425459 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:01:32.425526 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0619 09:01:32.425599 (+ 73us) server_negotiation.cc:299] Negotiation successful
0619 09:01:32.425650 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260619 09:01:32.427032 7921 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33932:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 09:01:32.428534 7921 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 09:01:32.434507 8423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:32.430404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41951 (local address 127.0.0.1:51328)
0619 09:01:32.430508 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0619 09:01:32.430540 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0619 09:01:32.430594 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:01:32.431383 (+ 789us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:01:32.431384 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:01:32.431386 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 09:01:32.431425 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:32.431426 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:32.432243 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:32.432244 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:32.433373 (+ 1129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:32.433375 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:32.434248 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:32.434249 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:32.434323 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:32.434338 (+ 15us) client_negotiation.cc:769] Sending connection context
0619 09:01:32.434400 (+ 62us) client_negotiation.cc:240] Negotiation successful
0619 09:01:32.434451 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":82,"thread_start_us":53,"threads_started":1}
I20260619 09:01:32.434752 8426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:32.430596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44393 (local address 127.0.0.1:38158)
0619 09:01:32.430894 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0619 09:01:32.430903 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 09:01:32.430950 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:01:32.431226 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:01:32.431230 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:01:32.431232 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 09:01:32.431292 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:32.431294 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:32.432618 (+ 1324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:32.432619 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:32.433589 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:32.433590 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:32.434506 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:32.434507 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:32.434581 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:32.434585 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 09:01:32.434632 (+ 47us) client_negotiation.cc:240] Negotiation successful
0619 09:01:32.434670 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":33,"threads_started":1}
I20260619 09:01:32.434839 8424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:32.430511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51328 (local address 127.0.98.2:41951)
0619 09:01:32.431189 (+ 678us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:32.431193 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:32.431202 (+ 9us) server_negotiation.cc:407] Connection header received
0619 09:01:32.431235 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:32.431237 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:32.431250 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:32.431319 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 09:01:32.431490 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:32.432167 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:32.433473 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:32.434166 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:32.434240 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:32.434316 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0619 09:01:32.434678 (+ 362us) server_negotiation.cc:299] Negotiation successful
0619 09:01:32.434720 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":647,"thread_start_us":101,"threads_started":1}
I20260619 09:01:32.435050 8427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:32.430706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38158 (local address 127.0.98.3:44393)
0619 09:01:32.430924 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:32.430929 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:32.430939 (+ 10us) server_negotiation.cc:407] Connection header received
0619 09:01:32.431025 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:32.431028 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:32.431043 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:32.431146 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 09:01:32.431417 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:32.432533 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:32.433676 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:32.434353 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:32.434410 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:32.434474 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0619 09:01:32.434919 (+ 445us) server_negotiation.cc:299] Negotiation successful
0619 09:01:32.434956 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":188,"thread_start_us":73,"threads_started":1}
I20260619 09:01:32.435575 8425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:32.430703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44937 (local address 127.0.0.1:48302)
0619 09:01:32.430997 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0619 09:01:32.431007 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 09:01:32.431066 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:01:32.431285 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:01:32.431286 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:01:32.431288 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 09:01:32.431342 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:32.431344 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:32.433653 (+ 2309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:32.433654 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:32.434326 (+ 672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:32.434328 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:32.435354 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:32.435355 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:32.435415 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:32.435419 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 09:01:32.435463 (+ 44us) client_negotiation.cc:240] Negotiation successful
0619 09:01:32.435501 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":63,"threads_started":1}
I20260619 09:01:32.435473 8198 tablet_service.cc:1467] Processing CreateTablet for tablet 9493b39090b24f23b801158a57932dd6 (DEFAULT_TABLE table=test-table [id=dcbad6fe95654d189846ca0f94dfcd80]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 09:01:32.435617 8338 tablet_service.cc:1467] Processing CreateTablet for tablet 9493b39090b24f23b801158a57932dd6 (DEFAULT_TABLE table=test-table [id=dcbad6fe95654d189846ca0f94dfcd80]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 09:01:32.435796 8428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:32.430842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48302 (local address 127.0.98.1:44937)
0619 09:01:32.430992 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:32.430997 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:32.431055 (+ 58us) server_negotiation.cc:407] Connection header received
0619 09:01:32.431140 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:32.431143 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:32.431158 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:32.431226 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 09:01:32.431415 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:32.432543 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:32.434402 (+ 1859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:32.435181 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:32.435244 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:32.435320 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0619 09:01:32.435662 (+ 342us) server_negotiation.cc:299] Negotiation successful
0619 09:01:32.435703 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":70,"threads_started":1}
I20260619 09:01:32.435832 8198 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9493b39090b24f23b801158a57932dd6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 09:01:32.435875 8338 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9493b39090b24f23b801158a57932dd6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 09:01:32.436275 8058 tablet_service.cc:1467] Processing CreateTablet for tablet 9493b39090b24f23b801158a57932dd6 (DEFAULT_TABLE table=test-table [id=dcbad6fe95654d189846ca0f94dfcd80]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 09:01:32.436506 8058 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9493b39090b24f23b801158a57932dd6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 09:01:32.439394 8429 tablet_bootstrap.cc:492] T 9493b39090b24f23b801158a57932dd6 P c9d6290881cd499dbaf9dd32d03583aa: Bootstrap starting.
I20260619 09:01:32.439486 8430 tablet_bootstrap.cc:492] T 9493b39090b24f23b801158a57932dd6 P 0269a239f4624610a1ce99d87e3f8967: Bootstrap starting.
I20260619 09:01:32.439954 8429 tablet_bootstrap.cc:654] T 9493b39090b24f23b801158a57932dd6 P c9d6290881cd499dbaf9dd32d03583aa: Neither blocks nor log segments found. Creating new log.
I20260619 09:01:32.440039 8431 tablet_bootstrap.cc:492] T 9493b39090b24f23b801158a57932dd6 P f06926dc12064dd78ea8de2e1a4259ed: Bootstrap starting.
I20260619 09:01:32.440121 8430 tablet_bootstrap.cc:654] T 9493b39090b24f23b801158a57932dd6 P 0269a239f4624610a1ce99d87e3f8967: Neither blocks nor log segments found. Creating new log.
I20260619 09:01:32.440279 8429 log.cc:824] T 9493b39090b24f23b801158a57932dd6 P c9d6290881cd499dbaf9dd32d03583aa: Log is configured to *not* fsync() on all Append() calls
I20260619 09:01:32.440456 8430 log.cc:824] T 9493b39090b24f23b801158a57932dd6 P 0269a239f4624610a1ce99d87e3f8967: Log is configured to *not* fsync() on all Append() calls
I20260619 09:01:32.440600 8431 tablet_bootstrap.cc:654] T 9493b39090b24f23b801158a57932dd6 P f06926dc12064dd78ea8de2e1a4259ed: Neither blocks nor log segments found. Creating new log.
I20260619 09:01:32.440964 8431 log.cc:824] T 9493b39090b24f23b801158a57932dd6 P f06926dc12064dd78ea8de2e1a4259ed: Log is configured to *not* fsync() on all Append() calls
I20260619 09:01:32.440954 8429 tablet_bootstrap.cc:492] T 9493b39090b24f23b801158a57932dd6 P c9d6290881cd499dbaf9dd32d03583aa: No bootstrap required, opened a new log
I20260619 09:01:32.441097 8429 ts_tablet_manager.cc:1397] T 9493b39090b24f23b801158a57932dd6 P c9d6290881cd499dbaf9dd32d03583aa: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 09:01:32.441227 8430 tablet_bootstrap.cc:492] T 9493b39090b24f23b801158a57932dd6 P 0269a239f4624610a1ce99d87e3f8967: No bootstrap required, opened a new log
I20260619 09:01:32.441287 8430 ts_tablet_manager.cc:1397] T 9493b39090b24f23b801158a57932dd6 P 0269a239f4624610a1ce99d87e3f8967: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 09:01:32.441592 8431 tablet_bootstrap.cc:492] T 9493b39090b24f23b801158a57932dd6 P f06926dc12064dd78ea8de2e1a4259ed: No bootstrap required, opened a new log
I20260619 09:01:32.441654 8431 ts_tablet_manager.cc:1397] T 9493b39090b24f23b801158a57932dd6 P f06926dc12064dd78ea8de2e1a4259ed: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 09:01:32.443112 8431 raft_consensus.cc:348] T 9493b39090b24f23b801158a57932dd6 P f06926dc12064dd78ea8de2e1a4259ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f06926dc12064dd78ea8de2e1a4259ed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41951 } } peers { permanent_uuid: "0269a239f4624610a1ce99d87e3f8967" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44393 } } peers { permanent_uuid: "c9d6290881cd499dbaf9dd32d03583aa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44937 } }
I20260619 09:01:32.443243 8431 raft_consensus.cc:374] T 9493b39090b24f23b801158a57932dd6 P f06926dc12064dd78ea8de2e1a4259ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 09:01:32.443272 8431 raft_consensus.cc:729] T 9493b39090b24f23b801158a57932dd6 P f06926dc12064dd78ea8de2e1a4259ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f06926dc12064dd78ea8de2e1a4259ed, State: Initialized, Role: FOLLOWER
I20260619 09:01:32.443218 8429 raft_consensus.cc:348] T 9493b39090b24f23b801158a57932dd6 P c9d6290881cd499dbaf9dd32d03583aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f06926dc12064dd78ea8de2e1a4259ed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41951 } } peers { permanent_uuid: "0269a239f4624610a1ce99d87e3f8967" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44393 } } peers { permanent_uuid: "c9d6290881cd499dbaf9dd32d03583aa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44937 } }
I20260619 09:01:32.443205 8430 raft_consensus.cc:348] T 9493b39090b24f23b801158a57932dd6 P 0269a239f4624610a1ce99d87e3f8967 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f06926dc12064dd78ea8de2e1a4259ed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41951 } } peers { permanent_uuid: "0269a239f4624610a1ce99d87e3f8967" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44393 } } peers { permanent_uuid: "c9d6290881cd499dbaf9dd32d03583aa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44937 } }
I20260619 09:01:32.443349 8430 raft_consensus.cc:374] T 9493b39090b24f23b801158a57932dd6 P 0269a239f4624610a1ce99d87e3f8967 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 09:01:32.443349 8429 raft_consensus.cc:374] T 9493b39090b24f23b801158a57932dd6 P c9d6290881cd499dbaf9dd32d03583aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 09:01:32.443389 8429 raft_consensus.cc:729] T 9493b39090b24f23b801158a57932dd6 P c9d6290881cd499dbaf9dd32d03583aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9d6290881cd499dbaf9dd32d03583aa, State: Initialized, Role: FOLLOWER
I20260619 09:01:32.443389 8430 raft_consensus.cc:729] T 9493b39090b24f23b801158a57932dd6 P 0269a239f4624610a1ce99d87e3f8967 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0269a239f4624610a1ce99d87e3f8967, State: Initialized, Role: FOLLOWER
I20260619 09:01:32.443400 8431 consensus_queue.cc:260] T 9493b39090b24f23b801158a57932dd6 P f06926dc12064dd78ea8de2e1a4259ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f06926dc12064dd78ea8de2e1a4259ed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41951 } } peers { permanent_uuid: "0269a239f4624610a1ce99d87e3f8967" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44393 } } peers { permanent_uuid: "c9d6290881cd499dbaf9dd32d03583aa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44937 } }
I20260619 09:01:32.443487 8430 consensus_queue.cc:260] T 9493b39090b24f23b801158a57932dd6 P 0269a239f4624610a1ce99d87e3f8967 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f06926dc12064dd78ea8de2e1a4259ed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41951 } } peers { permanent_uuid: "0269a239f4624610a1ce99d87e3f8967" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44393 } } peers { permanent_uuid: "c9d6290881cd499dbaf9dd32d03583aa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44937 } }
I20260619 09:01:32.443487 8429 consensus_queue.cc:260] T 9493b39090b24f23b801158a57932dd6 P c9d6290881cd499dbaf9dd32d03583aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f06926dc12064dd78ea8de2e1a4259ed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41951 } } peers { permanent_uuid: "0269a239f4624610a1ce99d87e3f8967" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44393 } } peers { permanent_uuid: "c9d6290881cd499dbaf9dd32d03583aa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44937 } }
I20260619 09:01:32.443650 8431 ts_tablet_manager.cc:1428] T 9493b39090b24f23b801158a57932dd6 P f06926dc12064dd78ea8de2e1a4259ed: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 09:01:32.443732 8429 ts_tablet_manager.cc:1428] T 9493b39090b24f23b801158a57932dd6 P c9d6290881cd499dbaf9dd32d03583aa: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260619 09:01:32.443732 8430 ts_tablet_manager.cc:1428] T 9493b39090b24f23b801158a57932dd6 P 0269a239f4624610a1ce99d87e3f8967: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260619 09:01:32.443770 8264 heartbeater.cc:503] Master 127.0.98.62:35301 was elected leader, sending a full tablet report...
I20260619 09:01:32.443840 8124 heartbeater.cc:503] Master 127.0.98.62:35301 was elected leader, sending a full tablet report...
I20260619 09:01:32.443961 8404 heartbeater.cc:503] Master 127.0.98.62:35301 was elected leader, sending a full tablet report...
W20260619 09:01:32.491856 8265 tablet.cc:2369] T 9493b39090b24f23b801158a57932dd6 P f06926dc12064dd78ea8de2e1a4259ed: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 09:01:32.560555 8437 raft_consensus.cc:482] T 9493b39090b24f23b801158a57932dd6 P c9d6290881cd499dbaf9dd32d03583aa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 09:01:32.560693 8437 raft_consensus.cc:504] T 9493b39090b24f23b801158a57932dd6 P c9d6290881cd499dbaf9dd32d03583aa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f06926dc12064dd78ea8de2e1a4259ed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41951 } } peers { permanent_uuid: "0269a239f4624610a1ce99d87e3f8967" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44393 } } peers { permanent_uuid: "c9d6290881cd499dbaf9dd32d03583aa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44937 } }
I20260619 09:01:32.561026 8437 leader_election.cc:288] T 9493b39090b24f23b801158a57932dd6 P c9d6290881cd499dbaf9dd32d03583aa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f06926dc12064dd78ea8de2e1a4259ed (127.0.98.2:41951), 0269a239f4624610a1ce99d87e3f8967 (127.0.98.3:44393)
I20260619 09:01:32.564738 8427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:32.561461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58343 (local address 127.0.98.3:44393)
0619 09:01:32.561503 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:32.561505 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:32.561575 (+ 70us) server_negotiation.cc:407] Connection header received
0619 09:01:32.561626 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:32.561627 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:32.561637 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:32.561687 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 09:01:32.561818 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:32.562503 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:32.563853 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:32.564403 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:32.564481 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:32.564493 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0619 09:01:32.564634 (+ 141us) server_negotiation.cc:299] Negotiation successful
0619 09:01:32.564673 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260619 09:01:32.564754 8424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:32.561324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57741 (local address 127.0.98.2:41951)
0619 09:01:32.561380 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:32.561382 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:32.561394 (+ 12us) server_negotiation.cc:407] Connection header received
0619 09:01:32.561427 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:32.561427 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:32.561439 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:32.561547 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 09:01:32.561696 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:32.562431 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:32.563853 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:32.564431 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:32.564536 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:32.564547 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0619 09:01:32.564663 (+ 116us) server_negotiation.cc:299] Negotiation successful
0619 09:01:32.564701 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260619 09:01:32.564766 8438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:32.561337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44393 (local address 127.0.98.1:58343)
0619 09:01:32.561505 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0619 09:01:32.561513 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0619 09:01:32.561570 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:01:32.561704 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:01:32.561705 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:01:32.561706 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 09:01:32.561762 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:32.561763 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:32.562571 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:32.562571 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:32.563762 (+ 1191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:32.563765 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:32.564486 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:32.564487 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:32.564566 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:32.564571 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 09:01:32.564624 (+ 53us) client_negotiation.cc:240] Negotiation successful
0619 09:01:32.564663 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"mutex_wait_us":2,"thread_start_us":120,"threads_started":1}
I20260619 09:01:32.564779 8017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:32.561225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41951 (local address 127.0.98.1:57741)
0619 09:01:32.561286 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0619 09:01:32.561298 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 09:01:32.561360 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:01:32.561552 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:01:32.561554 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:01:32.561555 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 09:01:32.561624 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:32.561626 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:32.562519 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:32.562520 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:32.563762 (+ 1242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:32.563765 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:32.564537 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:32.564537 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:32.564604 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:32.564606 (+ 2us) client_negotiation.cc:769] Sending connection context
0619 09:01:32.564651 (+ 45us) client_negotiation.cc:240] Negotiation successful
0619 09:01:32.564680 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38,"mutex_wait_us":10}
I20260619 09:01:32.565210 8218 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9493b39090b24f23b801158a57932dd6" candidate_uuid: "c9d6290881cd499dbaf9dd32d03583aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f06926dc12064dd78ea8de2e1a4259ed" is_pre_election: true
I20260619 09:01:32.565214 8358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9493b39090b24f23b801158a57932dd6" candidate_uuid: "c9d6290881cd499dbaf9dd32d03583aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0269a239f4624610a1ce99d87e3f8967" is_pre_election: true
I20260619 09:01:32.565359 8358 raft_consensus.cc:2454] T 9493b39090b24f23b801158a57932dd6 P 0269a239f4624610a1ce99d87e3f8967 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c9d6290881cd499dbaf9dd32d03583aa in term 0.
I20260619 09:01:32.565359 8218 raft_consensus.cc:2454] T 9493b39090b24f23b801158a57932dd6 P f06926dc12064dd78ea8de2e1a4259ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c9d6290881cd499dbaf9dd32d03583aa in term 0.
I20260619 09:01:32.565603 8011 leader_election.cc:302] T 9493b39090b24f23b801158a57932dd6 P c9d6290881cd499dbaf9dd32d03583aa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c9d6290881cd499dbaf9dd32d03583aa, f06926dc12064dd78ea8de2e1a4259ed; no voters:
I20260619 09:01:32.565734 8437 raft_consensus.cc:2781] T 9493b39090b24f23b801158a57932dd6 P c9d6290881cd499dbaf9dd32d03583aa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 09:01:32.565789 8437 raft_consensus.cc:482] T 9493b39090b24f23b801158a57932dd6 P c9d6290881cd499dbaf9dd32d03583aa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 09:01:32.565810 8437 raft_consensus.cc:3037] T 9493b39090b24f23b801158a57932dd6 P c9d6290881cd499dbaf9dd32d03583aa [term 0 FOLLOWER]: Advancing to term 1
I20260619 09:01:32.566696 8437 raft_consensus.cc:504] T 9493b39090b24f23b801158a57932dd6 P c9d6290881cd499dbaf9dd32d03583aa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f06926dc12064dd78ea8de2e1a4259ed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41951 } } peers { permanent_uuid: "0269a239f4624610a1ce99d87e3f8967" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44393 } } peers { permanent_uuid: "c9d6290881cd499dbaf9dd32d03583aa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44937 } }
I20260619 09:01:32.566805 8437 leader_election.cc:288] T 9493b39090b24f23b801158a57932dd6 P c9d6290881cd499dbaf9dd32d03583aa [CANDIDATE]: Term 1 election: Requested vote from peers f06926dc12064dd78ea8de2e1a4259ed (127.0.98.2:41951), 0269a239f4624610a1ce99d87e3f8967 (127.0.98.3:44393)
I20260619 09:01:32.567008 8358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9493b39090b24f23b801158a57932dd6" candidate_uuid: "c9d6290881cd499dbaf9dd32d03583aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0269a239f4624610a1ce99d87e3f8967"
I20260619 09:01:32.567080 8358 raft_consensus.cc:3037] T 9493b39090b24f23b801158a57932dd6 P 0269a239f4624610a1ce99d87e3f8967 [term 0 FOLLOWER]: Advancing to term 1
I20260619 09:01:32.567073 8218 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9493b39090b24f23b801158a57932dd6" candidate_uuid: "c9d6290881cd499dbaf9dd32d03583aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f06926dc12064dd78ea8de2e1a4259ed"
I20260619 09:01:32.567185 8218 raft_consensus.cc:3037] T 9493b39090b24f23b801158a57932dd6 P f06926dc12064dd78ea8de2e1a4259ed [term 0 FOLLOWER]: Advancing to term 1
I20260619 09:01:32.567754 8358 raft_consensus.cc:2454] T 9493b39090b24f23b801158a57932dd6 P 0269a239f4624610a1ce99d87e3f8967 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9d6290881cd499dbaf9dd32d03583aa in term 1.
I20260619 09:01:32.567900 8011 leader_election.cc:302] T 9493b39090b24f23b801158a57932dd6 P c9d6290881cd499dbaf9dd32d03583aa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0269a239f4624610a1ce99d87e3f8967, c9d6290881cd499dbaf9dd32d03583aa; no voters:
I20260619 09:01:32.567909 8218 raft_consensus.cc:2454] T 9493b39090b24f23b801158a57932dd6 P f06926dc12064dd78ea8de2e1a4259ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9d6290881cd499dbaf9dd32d03583aa in term 1.
I20260619 09:01:32.567977 8437 raft_consensus.cc:2781] T 9493b39090b24f23b801158a57932dd6 P c9d6290881cd499dbaf9dd32d03583aa [term 1 FOLLOWER]: Leader election won for term 1
I20260619 09:01:32.568115 8437 raft_consensus.cc:686] T 9493b39090b24f23b801158a57932dd6 P c9d6290881cd499dbaf9dd32d03583aa [term 1 LEADER]: Becoming Leader. State: Replica: c9d6290881cd499dbaf9dd32d03583aa, State: Running, Role: LEADER
I20260619 09:01:32.568212 8437 consensus_queue.cc:237] T 9493b39090b24f23b801158a57932dd6 P c9d6290881cd499dbaf9dd32d03583aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f06926dc12064dd78ea8de2e1a4259ed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41951 } } peers { permanent_uuid: "0269a239f4624610a1ce99d87e3f8967" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44393 } } peers { permanent_uuid: "c9d6290881cd499dbaf9dd32d03583aa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44937 } }
I20260619 09:01:32.569024 7919 catalog_manager.cc:5495] T 9493b39090b24f23b801158a57932dd6 P c9d6290881cd499dbaf9dd32d03583aa reported cstate change: term changed from 0 to 1, leader changed from <none> to c9d6290881cd499dbaf9dd32d03583aa (127.0.98.1). New cstate: current_term: 1 leader_uuid: "c9d6290881cd499dbaf9dd32d03583aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f06926dc12064dd78ea8de2e1a4259ed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41951 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0269a239f4624610a1ce99d87e3f8967" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44393 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c9d6290881cd499dbaf9dd32d03583aa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44937 } health_report { overall_health: HEALTHY } } }
I20260619 09:01:32.577698 8428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:32.575588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48312 (local address 127.0.98.1:44937)
0619 09:01:32.575635 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:32.575638 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:32.575645 (+ 7us) server_negotiation.cc:407] Connection header received
0619 09:01:32.575679 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:32.575680 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:32.575699 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:32.575760 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 09:01:32.575866 (+ 106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:32.576325 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:32.577030 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:32.577225 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:32.577278 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:32.577410 (+ 132us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 09:01:32.577507 (+ 97us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 09:01:32.577573 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0619 09:01:32.577605 (+ 32us) server_negotiation.cc:299] Negotiation successful
0619 09:01:32.577626 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260619 09:01:32.579416 8358 raft_consensus.cc:1261] T 9493b39090b24f23b801158a57932dd6 P 0269a239f4624610a1ce99d87e3f8967 [term 1 FOLLOWER]: Refusing update from remote peer c9d6290881cd499dbaf9dd32d03583aa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 09:01:32.579411 8218 raft_consensus.cc:1261] T 9493b39090b24f23b801158a57932dd6 P f06926dc12064dd78ea8de2e1a4259ed [term 1 FOLLOWER]: Refusing update from remote peer c9d6290881cd499dbaf9dd32d03583aa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 09:01:32.579721 8439 consensus_queue.cc:1035] T 9493b39090b24f23b801158a57932dd6 P c9d6290881cd499dbaf9dd32d03583aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "0269a239f4624610a1ce99d87e3f8967" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44393 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 09:01:32.579849 8437 consensus_queue.cc:1035] T 9493b39090b24f23b801158a57932dd6 P c9d6290881cd499dbaf9dd32d03583aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "f06926dc12064dd78ea8de2e1a4259ed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 09:01:32.582176 8442 mvcc.cc:205] Tried to move back new op lower bound from 7298497300802633728 to 7298497300760064000. Current Snapshot: MvccSnapshot[applied={T|T < 7298497300802633728}]
I20260619 09:01:32.583163 8443 mvcc.cc:205] Tried to move back new op lower bound from 7298497300802633728 to 7298497300760064000. Current Snapshot: MvccSnapshot[applied={T|T < 7298497300802633728}]
I20260619 09:01:32.584244 7919 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33932:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 09:01:32.584342 7919 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33932:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 09:01:32.585074 7919 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 75e0d761165f4e3f9aa72ac34214c70c: Sending DeleteTablet for 3 replicas of tablet 9493b39090b24f23b801158a57932dd6
I20260619 09:01:32.585484 8198 tablet_service.cc:1514] Processing DeleteTablet for tablet 9493b39090b24f23b801158a57932dd6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 09:01:32 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51328
I20260619 09:01:32.585482 8338 tablet_service.cc:1514] Processing DeleteTablet for tablet 9493b39090b24f23b801158a57932dd6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 09:01:32 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38158
I20260619 09:01:32.585536 8058 tablet_service.cc:1514] Processing DeleteTablet for tablet 9493b39090b24f23b801158a57932dd6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 09:01:32 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48302
I20260619 09:01:32.585795 8453 tablet_replica.cc:330] T 9493b39090b24f23b801158a57932dd6 P c9d6290881cd499dbaf9dd32d03583aa: stopping tablet replica
I20260619 09:01:32.585783 7919 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33932:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260619 09:01:32.585891 8453 raft_consensus.cc:2229] T 9493b39090b24f23b801158a57932dd6 P c9d6290881cd499dbaf9dd32d03583aa [term 1 LEADER]: Raft consensus shutting down.
W20260619 09:01:32.585906 7919 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 09:01:32.585950 8451 tablet_replica.cc:330] T 9493b39090b24f23b801158a57932dd6 P f06926dc12064dd78ea8de2e1a4259ed: stopping tablet replica
I20260619 09:01:32.586000 8453 raft_consensus.cc:2258] T 9493b39090b24f23b801158a57932dd6 P c9d6290881cd499dbaf9dd32d03583aa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 09:01:32.586021 8451 raft_consensus.cc:2229] T 9493b39090b24f23b801158a57932dd6 P f06926dc12064dd78ea8de2e1a4259ed [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 09:01:32.586105 8451 raft_consensus.cc:2258] T 9493b39090b24f23b801158a57932dd6 P f06926dc12064dd78ea8de2e1a4259ed [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 09:01:32.586251 8452 tablet_replica.cc:330] T 9493b39090b24f23b801158a57932dd6 P 0269a239f4624610a1ce99d87e3f8967: stopping tablet replica
I20260619 09:01:32.586338 8452 raft_consensus.cc:2229] T 9493b39090b24f23b801158a57932dd6 P 0269a239f4624610a1ce99d87e3f8967 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 09:01:32.586438 8452 raft_consensus.cc:2258] T 9493b39090b24f23b801158a57932dd6 P 0269a239f4624610a1ce99d87e3f8967 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 09:01:32.586925 8451 ts_tablet_manager.cc:1905] T 9493b39090b24f23b801158a57932dd6 P f06926dc12064dd78ea8de2e1a4259ed: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 09:01:32.587146 8453 ts_tablet_manager.cc:1905] T 9493b39090b24f23b801158a57932dd6 P c9d6290881cd499dbaf9dd32d03583aa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 09:01:32.587164 8198 tablet_service.cc:1467] Processing CreateTablet for tablet 86b6850e7c494e38985d8e3b55df01e5 (DEFAULT_TABLE table=test-table [id=ed9e1d37b96b4dcf9df55d4034f62081]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 09:01:32.587268 8198 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86b6850e7c494e38985d8e3b55df01e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 09:01:32.588546 8431 tablet_bootstrap.cc:492] T 86b6850e7c494e38985d8e3b55df01e5 P f06926dc12064dd78ea8de2e1a4259ed: Bootstrap starting.
I20260619 09:01:32.588702 8338 tablet_service.cc:1467] Processing CreateTablet for tablet 86b6850e7c494e38985d8e3b55df01e5 (DEFAULT_TABLE table=test-table [id=ed9e1d37b96b4dcf9df55d4034f62081]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 09:01:32.588800 8338 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86b6850e7c494e38985d8e3b55df01e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 09:01:32.588915 8431 tablet_bootstrap.cc:654] T 86b6850e7c494e38985d8e3b55df01e5 P f06926dc12064dd78ea8de2e1a4259ed: Neither blocks nor log segments found. Creating new log.
I20260619 09:01:32.589042 8453 ts_tablet_manager.cc:1918] T 9493b39090b24f23b801158a57932dd6 P c9d6290881cd499dbaf9dd32d03583aa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 09:01:32.589097 8453 log.cc:1196] T 9493b39090b24f23b801158a57932dd6 P c9d6290881cd499dbaf9dd32d03583aa: Deleting WAL directory at /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781859658134510-392-0/minicluster-data/ts-0/wal/wals/9493b39090b24f23b801158a57932dd6
I20260619 09:01:32.589233 8451 ts_tablet_manager.cc:1918] T 9493b39090b24f23b801158a57932dd6 P f06926dc12064dd78ea8de2e1a4259ed: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 09:01:32.589282 8451 log.cc:1196] T 9493b39090b24f23b801158a57932dd6 P f06926dc12064dd78ea8de2e1a4259ed: Deleting WAL directory at /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781859658134510-392-0/minicluster-data/ts-1/wal/wals/9493b39090b24f23b801158a57932dd6
I20260619 09:01:32.589314 8453 ts_tablet_manager.cc:1939] T 9493b39090b24f23b801158a57932dd6 P c9d6290881cd499dbaf9dd32d03583aa: Deleting consensus metadata
I20260619 09:01:32.589476 8451 ts_tablet_manager.cc:1939] T 9493b39090b24f23b801158a57932dd6 P f06926dc12064dd78ea8de2e1a4259ed: Deleting consensus metadata
I20260619 09:01:32.589893 8452 ts_tablet_manager.cc:1905] T 9493b39090b24f23b801158a57932dd6 P 0269a239f4624610a1ce99d87e3f8967: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 09:01:32.590185 8430 tablet_bootstrap.cc:492] T 86b6850e7c494e38985d8e3b55df01e5 P 0269a239f4624610a1ce99d87e3f8967: Bootstrap starting.
I20260619 09:01:32.590307 8058 tablet_service.cc:1467] Processing CreateTablet for tablet 86b6850e7c494e38985d8e3b55df01e5 (DEFAULT_TABLE table=test-table [id=ed9e1d37b96b4dcf9df55d4034f62081]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 09:01:32.590358 7907 catalog_manager.cc:4841] TS c9d6290881cd499dbaf9dd32d03583aa (127.0.98.1:44937): tablet 9493b39090b24f23b801158a57932dd6 (table test-table [id=dcbad6fe95654d189846ca0f94dfcd80]) successfully deleted
I20260619 09:01:32.590394 8058 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86b6850e7c494e38985d8e3b55df01e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 09:01:32.590431 7907 catalog_manager.cc:4841] TS f06926dc12064dd78ea8de2e1a4259ed (127.0.98.2:41951): tablet 9493b39090b24f23b801158a57932dd6 (table test-table [id=dcbad6fe95654d189846ca0f94dfcd80]) successfully deleted
I20260619 09:01:32.590548 8430 tablet_bootstrap.cc:654] T 86b6850e7c494e38985d8e3b55df01e5 P 0269a239f4624610a1ce99d87e3f8967: Neither blocks nor log segments found. Creating new log.
I20260619 09:01:32.591176 8430 tablet_bootstrap.cc:492] T 86b6850e7c494e38985d8e3b55df01e5 P 0269a239f4624610a1ce99d87e3f8967: No bootstrap required, opened a new log
I20260619 09:01:32.591235 8430 ts_tablet_manager.cc:1397] T 86b6850e7c494e38985d8e3b55df01e5 P 0269a239f4624610a1ce99d87e3f8967: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260619 09:01:32.591372 8430 raft_consensus.cc:348] T 86b6850e7c494e38985d8e3b55df01e5 P 0269a239f4624610a1ce99d87e3f8967 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9d6290881cd499dbaf9dd32d03583aa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44937 } } peers { permanent_uuid: "f06926dc12064dd78ea8de2e1a4259ed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41951 } } peers { permanent_uuid: "0269a239f4624610a1ce99d87e3f8967" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44393 } }
I20260619 09:01:32.591445 8430 raft_consensus.cc:374] T 86b6850e7c494e38985d8e3b55df01e5 P 0269a239f4624610a1ce99d87e3f8967 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 09:01:32.591441 8429 tablet_bootstrap.cc:492] T 86b6850e7c494e38985d8e3b55df01e5 P c9d6290881cd499dbaf9dd32d03583aa: Bootstrap starting.
I20260619 09:01:32.591465 8430 raft_consensus.cc:729] T 86b6850e7c494e38985d8e3b55df01e5 P 0269a239f4624610a1ce99d87e3f8967 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0269a239f4624610a1ce99d87e3f8967, State: Initialized, Role: FOLLOWER
I20260619 09:01:32.591476 8431 tablet_bootstrap.cc:492] T 86b6850e7c494e38985d8e3b55df01e5 P f06926dc12064dd78ea8de2e1a4259ed: No bootstrap required, opened a new log
I20260619 09:01:32.591516 8431 ts_tablet_manager.cc:1397] T 86b6850e7c494e38985d8e3b55df01e5 P f06926dc12064dd78ea8de2e1a4259ed: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260619 09:01:32.591502 8430 consensus_queue.cc:260] T 86b6850e7c494e38985d8e3b55df01e5 P 0269a239f4624610a1ce99d87e3f8967 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9d6290881cd499dbaf9dd32d03583aa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44937 } } peers { permanent_uuid: "f06926dc12064dd78ea8de2e1a4259ed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41951 } } peers { permanent_uuid: "0269a239f4624610a1ce99d87e3f8967" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44393 } }
I20260619 09:01:32.591579 8430 ts_tablet_manager.cc:1428] T 86b6850e7c494e38985d8e3b55df01e5 P 0269a239f4624610a1ce99d87e3f8967: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260619 09:01:32.591652 8431 raft_consensus.cc:348] T 86b6850e7c494e38985d8e3b55df01e5 P f06926dc12064dd78ea8de2e1a4259ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9d6290881cd499dbaf9dd32d03583aa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44937 } } peers { permanent_uuid: "f06926dc12064dd78ea8de2e1a4259ed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41951 } } peers { permanent_uuid: "0269a239f4624610a1ce99d87e3f8967" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44393 } }
I20260619 09:01:32.591710 8431 raft_consensus.cc:374] T 86b6850e7c494e38985d8e3b55df01e5 P f06926dc12064dd78ea8de2e1a4259ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 09:01:32.591729 8431 raft_consensus.cc:729] T 86b6850e7c494e38985d8e3b55df01e5 P f06926dc12064dd78ea8de2e1a4259ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f06926dc12064dd78ea8de2e1a4259ed, State: Initialized, Role: FOLLOWER
I20260619 09:01:32.591770 8431 consensus_queue.cc:260] T 86b6850e7c494e38985d8e3b55df01e5 P f06926dc12064dd78ea8de2e1a4259ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9d6290881cd499dbaf9dd32d03583aa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44937 } } peers { permanent_uuid: "f06926dc12064dd78ea8de2e1a4259ed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41951 } } peers { permanent_uuid: "0269a239f4624610a1ce99d87e3f8967" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44393 } }
I20260619 09:01:32.591809 8429 tablet_bootstrap.cc:654] T 86b6850e7c494e38985d8e3b55df01e5 P c9d6290881cd499dbaf9dd32d03583aa: Neither blocks nor log segments found. Creating new log.
I20260619 09:01:32.591842 8431 ts_tablet_manager.cc:1428] T 86b6850e7c494e38985d8e3b55df01e5 P f06926dc12064dd78ea8de2e1a4259ed: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260619 09:01:32.591181 8452 ts_tablet_manager.cc:1918] T 9493b39090b24f23b801158a57932dd6 P 0269a239f4624610a1ce99d87e3f8967: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 09:01:32.592089 8452 log.cc:1196] T 9493b39090b24f23b801158a57932dd6 P 0269a239f4624610a1ce99d87e3f8967: Deleting WAL directory at /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781859658134510-392-0/minicluster-data/ts-2/wal/wals/9493b39090b24f23b801158a57932dd6
I20260619 09:01:32.592312 8452 ts_tablet_manager.cc:1939] T 9493b39090b24f23b801158a57932dd6 P 0269a239f4624610a1ce99d87e3f8967: Deleting consensus metadata
I20260619 09:01:32.592527 8429 tablet_bootstrap.cc:492] T 86b6850e7c494e38985d8e3b55df01e5 P c9d6290881cd499dbaf9dd32d03583aa: No bootstrap required, opened a new log
I20260619 09:01:32.592576 8429 ts_tablet_manager.cc:1397] T 86b6850e7c494e38985d8e3b55df01e5 P c9d6290881cd499dbaf9dd32d03583aa: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260619 09:01:32.592660 7907 catalog_manager.cc:4841] TS 0269a239f4624610a1ce99d87e3f8967 (127.0.98.3:44393): tablet 9493b39090b24f23b801158a57932dd6 (table test-table [id=dcbad6fe95654d189846ca0f94dfcd80]) successfully deleted
I20260619 09:01:32.592723 8429 raft_consensus.cc:348] T 86b6850e7c494e38985d8e3b55df01e5 P c9d6290881cd499dbaf9dd32d03583aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9d6290881cd499dbaf9dd32d03583aa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44937 } } peers { permanent_uuid: "f06926dc12064dd78ea8de2e1a4259ed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41951 } } peers { permanent_uuid: "0269a239f4624610a1ce99d87e3f8967" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44393 } }
I20260619 09:01:32.592790 8429 raft_consensus.cc:374] T 86b6850e7c494e38985d8e3b55df01e5 P c9d6290881cd499dbaf9dd32d03583aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 09:01:32.592818 8429 raft_consensus.cc:729] T 86b6850e7c494e38985d8e3b55df01e5 P c9d6290881cd499dbaf9dd32d03583aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9d6290881cd499dbaf9dd32d03583aa, State: Initialized, Role: FOLLOWER
I20260619 09:01:32.592882 8429 consensus_queue.cc:260] T 86b6850e7c494e38985d8e3b55df01e5 P c9d6290881cd499dbaf9dd32d03583aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9d6290881cd499dbaf9dd32d03583aa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44937 } } peers { permanent_uuid: "f06926dc12064dd78ea8de2e1a4259ed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41951 } } peers { permanent_uuid: "0269a239f4624610a1ce99d87e3f8967" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44393 } }
I20260619 09:01:32.592962 8429 ts_tablet_manager.cc:1428] T 86b6850e7c494e38985d8e3b55df01e5 P c9d6290881cd499dbaf9dd32d03583aa: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
W20260619 09:01:32.594365 8125 tablet.cc:2369] T 86b6850e7c494e38985d8e3b55df01e5 P c9d6290881cd499dbaf9dd32d03583aa: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 09:01:32.640915 8405 tablet.cc:2369] T 86b6850e7c494e38985d8e3b55df01e5 P 0269a239f4624610a1ce99d87e3f8967: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 09:01:32.874459 8436 raft_consensus.cc:482] T 86b6850e7c494e38985d8e3b55df01e5 P 0269a239f4624610a1ce99d87e3f8967 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 09:01:32.874550 8436 raft_consensus.cc:504] T 86b6850e7c494e38985d8e3b55df01e5 P 0269a239f4624610a1ce99d87e3f8967 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9d6290881cd499dbaf9dd32d03583aa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44937 } } peers { permanent_uuid: "f06926dc12064dd78ea8de2e1a4259ed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41951 } } peers { permanent_uuid: "0269a239f4624610a1ce99d87e3f8967" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44393 } }
I20260619 09:01:32.874848 8436 leader_election.cc:288] T 86b6850e7c494e38985d8e3b55df01e5 P 0269a239f4624610a1ce99d87e3f8967 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c9d6290881cd499dbaf9dd32d03583aa (127.0.98.1:44937), f06926dc12064dd78ea8de2e1a4259ed (127.0.98.2:41951)
I20260619 09:01:32.877573 8428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:32.875066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60721 (local address 127.0.98.1:44937)
0619 09:01:32.875113 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:32.875115 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:32.875128 (+ 13us) server_negotiation.cc:407] Connection header received
0619 09:01:32.875160 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:32.875161 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:32.875169 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:32.875246 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 09:01:32.875404 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:32.875950 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:32.876882 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:32.877261 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:32.877311 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:32.877320 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0619 09:01:32.877460 (+ 140us) server_negotiation.cc:299] Negotiation successful
0619 09:01:32.877502 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260619 09:01:32.877566 8297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:32.875006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44937 (local address 127.0.98.3:60721)
0619 09:01:32.875052 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0619 09:01:32.875064 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 09:01:32.875108 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:01:32.875251 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:01:32.875251 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:01:32.875253 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 09:01:32.875332 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:32.875333 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:32.876022 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:32.876023 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:32.876801 (+ 778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:32.876803 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:32.877329 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:32.877330 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:32.877398 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:32.877402 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 09:01:32.877448 (+ 46us) client_negotiation.cc:240] Negotiation successful
0619 09:01:32.877481 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24}
I20260619 09:01:32.877637 8424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:32.875261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40421 (local address 127.0.98.2:41951)
0619 09:01:32.875301 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:32.875303 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:32.875311 (+ 8us) server_negotiation.cc:407] Connection header received
0619 09:01:32.875335 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:32.875335 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:32.875343 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:32.875403 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 09:01:32.875543 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:32.876049 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:32.876962 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:32.877325 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:32.877388 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:32.877398 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0619 09:01:32.877546 (+ 148us) server_negotiation.cc:299] Negotiation successful
0619 09:01:32.877582 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260619 09:01:32.877691 8457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:32.875100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41951 (local address 127.0.98.3:40421)
0619 09:01:32.875216 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0619 09:01:32.875226 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 09:01:32.875287 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:01:32.875412 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:01:32.875413 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:01:32.875414 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 09:01:32.875488 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:32.875490 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:32.876123 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:32.876124 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:32.876876 (+ 752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:32.876878 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:32.877411 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:32.877412 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:32.877474 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:32.877479 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 09:01:32.877529 (+ 50us) client_negotiation.cc:240] Negotiation successful
0619 09:01:32.877565 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":66,"threads_started":1}
I20260619 09:01:32.877947 8218 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86b6850e7c494e38985d8e3b55df01e5" candidate_uuid: "0269a239f4624610a1ce99d87e3f8967" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f06926dc12064dd78ea8de2e1a4259ed" is_pre_election: true
I20260619 09:01:32.877967 8078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86b6850e7c494e38985d8e3b55df01e5" candidate_uuid: "0269a239f4624610a1ce99d87e3f8967" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9d6290881cd499dbaf9dd32d03583aa" is_pre_election: true
I20260619 09:01:32.878036 8218 raft_consensus.cc:2454] T 86b6850e7c494e38985d8e3b55df01e5 P f06926dc12064dd78ea8de2e1a4259ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0269a239f4624610a1ce99d87e3f8967 in term 0.
I20260619 09:01:32.878086 8078 raft_consensus.cc:2454] T 86b6850e7c494e38985d8e3b55df01e5 P c9d6290881cd499dbaf9dd32d03583aa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0269a239f4624610a1ce99d87e3f8967 in term 0.
I20260619 09:01:32.878192 8291 leader_election.cc:302] T 86b6850e7c494e38985d8e3b55df01e5 P 0269a239f4624610a1ce99d87e3f8967 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0269a239f4624610a1ce99d87e3f8967, f06926dc12064dd78ea8de2e1a4259ed; no voters:
I20260619 09:01:32.878304 8436 raft_consensus.cc:2781] T 86b6850e7c494e38985d8e3b55df01e5 P 0269a239f4624610a1ce99d87e3f8967 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 09:01:32.878335 8436 raft_consensus.cc:482] T 86b6850e7c494e38985d8e3b55df01e5 P 0269a239f4624610a1ce99d87e3f8967 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 09:01:32.878347 8436 raft_consensus.cc:3037] T 86b6850e7c494e38985d8e3b55df01e5 P 0269a239f4624610a1ce99d87e3f8967 [term 0 FOLLOWER]: Advancing to term 1
I20260619 09:01:32.879067 8436 raft_consensus.cc:504] T 86b6850e7c494e38985d8e3b55df01e5 P 0269a239f4624610a1ce99d87e3f8967 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9d6290881cd499dbaf9dd32d03583aa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44937 } } peers { permanent_uuid: "f06926dc12064dd78ea8de2e1a4259ed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41951 } } peers { permanent_uuid: "0269a239f4624610a1ce99d87e3f8967" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44393 } }
I20260619 09:01:32.879175 8436 leader_election.cc:288] T 86b6850e7c494e38985d8e3b55df01e5 P 0269a239f4624610a1ce99d87e3f8967 [CANDIDATE]: Term 1 election: Requested vote from peers c9d6290881cd499dbaf9dd32d03583aa (127.0.98.1:44937), f06926dc12064dd78ea8de2e1a4259ed (127.0.98.2:41951)
I20260619 09:01:32.879335 8078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86b6850e7c494e38985d8e3b55df01e5" candidate_uuid: "0269a239f4624610a1ce99d87e3f8967" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9d6290881cd499dbaf9dd32d03583aa"
I20260619 09:01:32.879361 8218 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86b6850e7c494e38985d8e3b55df01e5" candidate_uuid: "0269a239f4624610a1ce99d87e3f8967" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f06926dc12064dd78ea8de2e1a4259ed"
I20260619 09:01:32.879422 8078 raft_consensus.cc:3037] T 86b6850e7c494e38985d8e3b55df01e5 P c9d6290881cd499dbaf9dd32d03583aa [term 0 FOLLOWER]: Advancing to term 1
I20260619 09:01:32.879426 8218 raft_consensus.cc:3037] T 86b6850e7c494e38985d8e3b55df01e5 P f06926dc12064dd78ea8de2e1a4259ed [term 0 FOLLOWER]: Advancing to term 1
I20260619 09:01:32.880013 8078 raft_consensus.cc:2454] T 86b6850e7c494e38985d8e3b55df01e5 P c9d6290881cd499dbaf9dd32d03583aa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0269a239f4624610a1ce99d87e3f8967 in term 1.
I20260619 09:01:32.880018 8218 raft_consensus.cc:2454] T 86b6850e7c494e38985d8e3b55df01e5 P f06926dc12064dd78ea8de2e1a4259ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0269a239f4624610a1ce99d87e3f8967 in term 1.
I20260619 09:01:32.880180 8291 leader_election.cc:302] T 86b6850e7c494e38985d8e3b55df01e5 P 0269a239f4624610a1ce99d87e3f8967 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0269a239f4624610a1ce99d87e3f8967, c9d6290881cd499dbaf9dd32d03583aa; no voters:
I20260619 09:01:32.880276 8436 raft_consensus.cc:2781] T 86b6850e7c494e38985d8e3b55df01e5 P 0269a239f4624610a1ce99d87e3f8967 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 09:01:32.880390 8436 raft_consensus.cc:686] T 86b6850e7c494e38985d8e3b55df01e5 P 0269a239f4624610a1ce99d87e3f8967 [term 1 LEADER]: Becoming Leader. State: Replica: 0269a239f4624610a1ce99d87e3f8967, State: Running, Role: LEADER
I20260619 09:01:32.880456 8436 consensus_queue.cc:237] T 86b6850e7c494e38985d8e3b55df01e5 P 0269a239f4624610a1ce99d87e3f8967 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9d6290881cd499dbaf9dd32d03583aa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44937 } } peers { permanent_uuid: "f06926dc12064dd78ea8de2e1a4259ed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41951 } } peers { permanent_uuid: "0269a239f4624610a1ce99d87e3f8967" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44393 } }
I20260619 09:01:32.881125 7919 catalog_manager.cc:5495] T 86b6850e7c494e38985d8e3b55df01e5 P 0269a239f4624610a1ce99d87e3f8967 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0269a239f4624610a1ce99d87e3f8967 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "0269a239f4624610a1ce99d87e3f8967" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9d6290881cd499dbaf9dd32d03583aa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44937 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f06926dc12064dd78ea8de2e1a4259ed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41951 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0269a239f4624610a1ce99d87e3f8967" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44393 } health_report { overall_health: HEALTHY } } }
Jun 19 09:01:32 dist-test-slave-mtdr krb5kdc[7876](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859691, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260619 09:01:32.962034 8466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:32.955588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33948 (local address 127.0.98.62:35301)
0619 09:01:32.955722 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:32.955724 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:32.955784 (+ 60us) server_negotiation.cc:407] Connection header received
0619 09:01:32.955851 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:32.955851 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:32.955861 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:32.955916 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:01:32.956360 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:32.956881 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:32.957615 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:32.957839 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:32.957897 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:32.959777 (+ 1880us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:01:32.959778 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:01:32.959779 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:01:32.959785 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:01:32.960995 (+ 1210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:01:32.961353 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:01:32.961354 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:01:32.961355 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:01:32.961412 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:01:32.961566 (+ 154us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:01:32.961566 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:01:32.961567 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:01:32.961666 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:01:32.961712 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0619 09:01:32.961885 (+ 173us) server_negotiation.cc:299] Negotiation successful
0619 09:01:32.961962 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":77,"threads_started":1}
I20260619 09:01:32.962145 8465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:32.955532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35301 (local address 127.0.0.1:33948)
0619 09:01:32.955712 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0619 09:01:32.955723 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 09:01:32.955792 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:01:32.955934 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:01:32.955935 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:01:32.956220 (+ 285us) client_negotiation.cc:189] Negotiated authn=SASL
0619 09:01:32.956268 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:32.956270 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:32.956974 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:32.956975 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:32.957527 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:32.957528 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:32.957898 (+ 370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:32.957898 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:32.957939 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:32.958382 (+ 443us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 09:01:32.958386 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 09:01:32.959687 (+ 1301us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 09:01:32.961076 (+ 1389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:01:32.961082 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:01:32.961084 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:01:32.961279 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:01:32.961478 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:01:32.961478 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:01:32.961479 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:01:32.961526 (+ 47us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:01:32.961742 (+ 216us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 09:01:32.961749 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 09:01:32.961797 (+ 48us) client_negotiation.cc:769] Sending connection context
0619 09:01:32.961866 (+ 69us) client_negotiation.cc:240] Negotiation successful
0619 09:01:32.962054 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"mutex_wait_us":83,"thread_start_us":76,"threads_started":1}
I20260619 09:01:32.963864 7919 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33948:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260619 09:01:32.964063 7919 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 09:01:32.965380 8338 tablet_service.cc:1467] Processing CreateTablet for tablet 19a175c688d54d8fae2aa105f1fca9cb (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=5f5a63c675b64203a8d8c52a2a59083c]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 09:01:32.965389 8198 tablet_service.cc:1467] Processing CreateTablet for tablet 19a175c688d54d8fae2aa105f1fca9cb (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=5f5a63c675b64203a8d8c52a2a59083c]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 09:01:32.965380 8058 tablet_service.cc:1467] Processing CreateTablet for tablet 19a175c688d54d8fae2aa105f1fca9cb (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=5f5a63c675b64203a8d8c52a2a59083c]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 09:01:32.965543 8198 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 19a175c688d54d8fae2aa105f1fca9cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 09:01:32.965543 8058 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 19a175c688d54d8fae2aa105f1fca9cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 09:01:32.965543 8338 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 19a175c688d54d8fae2aa105f1fca9cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 09:01:32.966672 8431 tablet_bootstrap.cc:492] T 19a175c688d54d8fae2aa105f1fca9cb P f06926dc12064dd78ea8de2e1a4259ed: Bootstrap starting.
I20260619 09:01:32.966948 8429 tablet_bootstrap.cc:492] T 19a175c688d54d8fae2aa105f1fca9cb P c9d6290881cd499dbaf9dd32d03583aa: Bootstrap starting.
I20260619 09:01:32.967109 8431 tablet_bootstrap.cc:654] T 19a175c688d54d8fae2aa105f1fca9cb P f06926dc12064dd78ea8de2e1a4259ed: Neither blocks nor log segments found. Creating new log.
I20260619 09:01:32.967257 8429 tablet_bootstrap.cc:654] T 19a175c688d54d8fae2aa105f1fca9cb P c9d6290881cd499dbaf9dd32d03583aa: Neither blocks nor log segments found. Creating new log.
I20260619 09:01:32.967449 8430 tablet_bootstrap.cc:492] T 19a175c688d54d8fae2aa105f1fca9cb P 0269a239f4624610a1ce99d87e3f8967: Bootstrap starting.
I20260619 09:01:32.967711 8431 tablet_bootstrap.cc:492] T 19a175c688d54d8fae2aa105f1fca9cb P f06926dc12064dd78ea8de2e1a4259ed: No bootstrap required, opened a new log
I20260619 09:01:32.967760 8431 ts_tablet_manager.cc:1397] T 19a175c688d54d8fae2aa105f1fca9cb P f06926dc12064dd78ea8de2e1a4259ed: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260619 09:01:32.967794 8430 tablet_bootstrap.cc:654] T 19a175c688d54d8fae2aa105f1fca9cb P 0269a239f4624610a1ce99d87e3f8967: Neither blocks nor log segments found. Creating new log.
I20260619 09:01:32.967912 8431 raft_consensus.cc:348] T 19a175c688d54d8fae2aa105f1fca9cb P f06926dc12064dd78ea8de2e1a4259ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9d6290881cd499dbaf9dd32d03583aa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44937 } } peers { permanent_uuid: "f06926dc12064dd78ea8de2e1a4259ed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41951 } } peers { permanent_uuid: "0269a239f4624610a1ce99d87e3f8967" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44393 } }
I20260619 09:01:32.968000 8431 raft_consensus.cc:374] T 19a175c688d54d8fae2aa105f1fca9cb P f06926dc12064dd78ea8de2e1a4259ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 09:01:32.968030 8431 raft_consensus.cc:729] T 19a175c688d54d8fae2aa105f1fca9cb P f06926dc12064dd78ea8de2e1a4259ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f06926dc12064dd78ea8de2e1a4259ed, State: Initialized, Role: FOLLOWER
I20260619 09:01:32.968078 8431 consensus_queue.cc:260] T 19a175c688d54d8fae2aa105f1fca9cb P f06926dc12064dd78ea8de2e1a4259ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9d6290881cd499dbaf9dd32d03583aa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44937 } } peers { permanent_uuid: "f06926dc12064dd78ea8de2e1a4259ed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41951 } } peers { permanent_uuid: "0269a239f4624610a1ce99d87e3f8967" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44393 } }
I20260619 09:01:32.968174 8431 ts_tablet_manager.cc:1428] T 19a175c688d54d8fae2aa105f1fca9cb P f06926dc12064dd78ea8de2e1a4259ed: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260619 09:01:32.968319 8435 tablet_replica.cc:439] T 19a175c688d54d8fae2aa105f1fca9cb P f06926dc12064dd78ea8de2e1a4259ed: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9d6290881cd499dbaf9dd32d03583aa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44937 } } peers { permanent_uuid: "f06926dc12064dd78ea8de2e1a4259ed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41951 } } peers { permanent_uuid: "0269a239f4624610a1ce99d87e3f8967" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44393 } } }
I20260619 09:01:32.968427 8435 tablet_replica.cc:442] T 19a175c688d54d8fae2aa105f1fca9cb P f06926dc12064dd78ea8de2e1a4259ed: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 09:01:32.968497 8430 tablet_bootstrap.cc:492] T 19a175c688d54d8fae2aa105f1fca9cb P 0269a239f4624610a1ce99d87e3f8967: No bootstrap required, opened a new log
I20260619 09:01:32.968534 8430 ts_tablet_manager.cc:1397] T 19a175c688d54d8fae2aa105f1fca9cb P 0269a239f4624610a1ce99d87e3f8967: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260619 09:01:32.968621 8429 tablet_bootstrap.cc:492] T 19a175c688d54d8fae2aa105f1fca9cb P c9d6290881cd499dbaf9dd32d03583aa: No bootstrap required, opened a new log
I20260619 09:01:32.968653 8429 ts_tablet_manager.cc:1397] T 19a175c688d54d8fae2aa105f1fca9cb P c9d6290881cd499dbaf9dd32d03583aa: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 09:01:32.968660 8430 raft_consensus.cc:348] T 19a175c688d54d8fae2aa105f1fca9cb P 0269a239f4624610a1ce99d87e3f8967 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9d6290881cd499dbaf9dd32d03583aa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44937 } } peers { permanent_uuid: "f06926dc12064dd78ea8de2e1a4259ed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41951 } } peers { permanent_uuid: "0269a239f4624610a1ce99d87e3f8967" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44393 } }
I20260619 09:01:32.968729 8430 raft_consensus.cc:374] T 19a175c688d54d8fae2aa105f1fca9cb P 0269a239f4624610a1ce99d87e3f8967 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 09:01:32.968752 8430 raft_consensus.cc:729] T 19a175c688d54d8fae2aa105f1fca9cb P 0269a239f4624610a1ce99d87e3f8967 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0269a239f4624610a1ce99d87e3f8967, State: Initialized, Role: FOLLOWER
I20260619 09:01:32.968755 8429 raft_consensus.cc:348] T 19a175c688d54d8fae2aa105f1fca9cb P c9d6290881cd499dbaf9dd32d03583aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9d6290881cd499dbaf9dd32d03583aa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44937 } } peers { permanent_uuid: "f06926dc12064dd78ea8de2e1a4259ed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41951 } } peers { permanent_uuid: "0269a239f4624610a1ce99d87e3f8967" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44393 } }
I20260619 09:01:32.968791 8430 consensus_queue.cc:260] T 19a175c688d54d8fae2aa105f1fca9cb P 0269a239f4624610a1ce99d87e3f8967 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9d6290881cd499dbaf9dd32d03583aa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44937 } } peers { permanent_uuid: "f06926dc12064dd78ea8de2e1a4259ed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41951 } } peers { permanent_uuid: "0269a239f4624610a1ce99d87e3f8967" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44393 } }
I20260619 09:01:32.968822 8429 raft_consensus.cc:374] T 19a175c688d54d8fae2aa105f1fca9cb P c9d6290881cd499dbaf9dd32d03583aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 09:01:32.968847 8429 raft_consensus.cc:729] T 19a175c688d54d8fae2aa105f1fca9cb P c9d6290881cd499dbaf9dd32d03583aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9d6290881cd499dbaf9dd32d03583aa, State: Initialized, Role: FOLLOWER
I20260619 09:01:32.968870 8430 ts_tablet_manager.cc:1428] T 19a175c688d54d8fae2aa105f1fca9cb P 0269a239f4624610a1ce99d87e3f8967: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260619 09:01:32.968883 8429 consensus_queue.cc:260] T 19a175c688d54d8fae2aa105f1fca9cb P c9d6290881cd499dbaf9dd32d03583aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9d6290881cd499dbaf9dd32d03583aa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44937 } } peers { permanent_uuid: "f06926dc12064dd78ea8de2e1a4259ed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41951 } } peers { permanent_uuid: "0269a239f4624610a1ce99d87e3f8967" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44393 } }
I20260619 09:01:32.968976 8429 ts_tablet_manager.cc:1428] T 19a175c688d54d8fae2aa105f1fca9cb P c9d6290881cd499dbaf9dd32d03583aa: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260619 09:01:32.968976 8436 tablet_replica.cc:439] T 19a175c688d54d8fae2aa105f1fca9cb P 0269a239f4624610a1ce99d87e3f8967: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9d6290881cd499dbaf9dd32d03583aa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44937 } } peers { permanent_uuid: "f06926dc12064dd78ea8de2e1a4259ed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41951 } } peers { permanent_uuid: "0269a239f4624610a1ce99d87e3f8967" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44393 } } }
I20260619 09:01:32.969043 8436 tablet_replica.cc:442] T 19a175c688d54d8fae2aa105f1fca9cb P 0269a239f4624610a1ce99d87e3f8967: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 09:01:32.969152 8437 tablet_replica.cc:439] T 19a175c688d54d8fae2aa105f1fca9cb P c9d6290881cd499dbaf9dd32d03583aa: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9d6290881cd499dbaf9dd32d03583aa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44937 } } peers { permanent_uuid: "f06926dc12064dd78ea8de2e1a4259ed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41951 } } peers { permanent_uuid: "0269a239f4624610a1ce99d87e3f8967" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44393 } } }
I20260619 09:01:32.969224 8437 tablet_replica.cc:442] T 19a175c688d54d8fae2aa105f1fca9cb P c9d6290881cd499dbaf9dd32d03583aa: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 09:01:33.090679 8437 raft_consensus.cc:482] T 19a175c688d54d8fae2aa105f1fca9cb P c9d6290881cd499dbaf9dd32d03583aa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 09:01:33.090772 8437 raft_consensus.cc:504] T 19a175c688d54d8fae2aa105f1fca9cb P c9d6290881cd499dbaf9dd32d03583aa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9d6290881cd499dbaf9dd32d03583aa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44937 } } peers { permanent_uuid: "f06926dc12064dd78ea8de2e1a4259ed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41951 } } peers { permanent_uuid: "0269a239f4624610a1ce99d87e3f8967" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44393 } }
I20260619 09:01:33.090920 8437 leader_election.cc:288] T 19a175c688d54d8fae2aa105f1fca9cb P c9d6290881cd499dbaf9dd32d03583aa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f06926dc12064dd78ea8de2e1a4259ed (127.0.98.2:41951), 0269a239f4624610a1ce99d87e3f8967 (127.0.98.3:44393)
I20260619 09:01:33.091158 8218 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19a175c688d54d8fae2aa105f1fca9cb" candidate_uuid: "c9d6290881cd499dbaf9dd32d03583aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f06926dc12064dd78ea8de2e1a4259ed" is_pre_election: true
I20260619 09:01:33.091158 8358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19a175c688d54d8fae2aa105f1fca9cb" candidate_uuid: "c9d6290881cd499dbaf9dd32d03583aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0269a239f4624610a1ce99d87e3f8967" is_pre_election: true
I20260619 09:01:33.091270 8218 raft_consensus.cc:2454] T 19a175c688d54d8fae2aa105f1fca9cb P f06926dc12064dd78ea8de2e1a4259ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c9d6290881cd499dbaf9dd32d03583aa in term 0.
I20260619 09:01:33.091270 8358 raft_consensus.cc:2454] T 19a175c688d54d8fae2aa105f1fca9cb P 0269a239f4624610a1ce99d87e3f8967 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c9d6290881cd499dbaf9dd32d03583aa in term 0.
I20260619 09:01:33.091437 8011 leader_election.cc:302] T 19a175c688d54d8fae2aa105f1fca9cb P c9d6290881cd499dbaf9dd32d03583aa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c9d6290881cd499dbaf9dd32d03583aa, f06926dc12064dd78ea8de2e1a4259ed; no voters:
I20260619 09:01:33.091576 8437 raft_consensus.cc:2781] T 19a175c688d54d8fae2aa105f1fca9cb P c9d6290881cd499dbaf9dd32d03583aa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 09:01:33.091622 8437 raft_consensus.cc:482] T 19a175c688d54d8fae2aa105f1fca9cb P c9d6290881cd499dbaf9dd32d03583aa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 09:01:33.091642 8437 raft_consensus.cc:3037] T 19a175c688d54d8fae2aa105f1fca9cb P c9d6290881cd499dbaf9dd32d03583aa [term 0 FOLLOWER]: Advancing to term 1
I20260619 09:01:33.092268 8437 raft_consensus.cc:504] T 19a175c688d54d8fae2aa105f1fca9cb P c9d6290881cd499dbaf9dd32d03583aa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9d6290881cd499dbaf9dd32d03583aa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44937 } } peers { permanent_uuid: "f06926dc12064dd78ea8de2e1a4259ed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41951 } } peers { permanent_uuid: "0269a239f4624610a1ce99d87e3f8967" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44393 } }
I20260619 09:01:33.092383 8437 leader_election.cc:288] T 19a175c688d54d8fae2aa105f1fca9cb P c9d6290881cd499dbaf9dd32d03583aa [CANDIDATE]: Term 1 election: Requested vote from peers f06926dc12064dd78ea8de2e1a4259ed (127.0.98.2:41951), 0269a239f4624610a1ce99d87e3f8967 (127.0.98.3:44393)
I20260619 09:01:33.092552 8358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19a175c688d54d8fae2aa105f1fca9cb" candidate_uuid: "c9d6290881cd499dbaf9dd32d03583aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0269a239f4624610a1ce99d87e3f8967"
I20260619 09:01:33.092555 8218 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19a175c688d54d8fae2aa105f1fca9cb" candidate_uuid: "c9d6290881cd499dbaf9dd32d03583aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f06926dc12064dd78ea8de2e1a4259ed"
I20260619 09:01:33.092621 8358 raft_consensus.cc:3037] T 19a175c688d54d8fae2aa105f1fca9cb P 0269a239f4624610a1ce99d87e3f8967 [term 0 FOLLOWER]: Advancing to term 1
I20260619 09:01:33.092621 8218 raft_consensus.cc:3037] T 19a175c688d54d8fae2aa105f1fca9cb P f06926dc12064dd78ea8de2e1a4259ed [term 0 FOLLOWER]: Advancing to term 1
I20260619 09:01:33.093295 8358 raft_consensus.cc:2454] T 19a175c688d54d8fae2aa105f1fca9cb P 0269a239f4624610a1ce99d87e3f8967 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9d6290881cd499dbaf9dd32d03583aa in term 1.
I20260619 09:01:33.093310 8218 raft_consensus.cc:2454] T 19a175c688d54d8fae2aa105f1fca9cb P f06926dc12064dd78ea8de2e1a4259ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9d6290881cd499dbaf9dd32d03583aa in term 1.
I20260619 09:01:33.093447 8011 leader_election.cc:302] T 19a175c688d54d8fae2aa105f1fca9cb P c9d6290881cd499dbaf9dd32d03583aa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c9d6290881cd499dbaf9dd32d03583aa, f06926dc12064dd78ea8de2e1a4259ed; no voters:
I20260619 09:01:33.093554 8437 raft_consensus.cc:2781] T 19a175c688d54d8fae2aa105f1fca9cb P c9d6290881cd499dbaf9dd32d03583aa [term 1 FOLLOWER]: Leader election won for term 1
I20260619 09:01:33.093647 8437 raft_consensus.cc:686] T 19a175c688d54d8fae2aa105f1fca9cb P c9d6290881cd499dbaf9dd32d03583aa [term 1 LEADER]: Becoming Leader. State: Replica: c9d6290881cd499dbaf9dd32d03583aa, State: Running, Role: LEADER
I20260619 09:01:33.093698 8437 consensus_queue.cc:237] T 19a175c688d54d8fae2aa105f1fca9cb P c9d6290881cd499dbaf9dd32d03583aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9d6290881cd499dbaf9dd32d03583aa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44937 } } peers { permanent_uuid: "f06926dc12064dd78ea8de2e1a4259ed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41951 } } peers { permanent_uuid: "0269a239f4624610a1ce99d87e3f8967" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44393 } }
I20260619 09:01:33.093999 8470 tablet_replica.cc:439] T 19a175c688d54d8fae2aa105f1fca9cb P c9d6290881cd499dbaf9dd32d03583aa: TxnStatusTablet state changed. Reason: New leader c9d6290881cd499dbaf9dd32d03583aa. Latest consensus state: current_term: 1 leader_uuid: "c9d6290881cd499dbaf9dd32d03583aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9d6290881cd499dbaf9dd32d03583aa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44937 } } peers { permanent_uuid: "f06926dc12064dd78ea8de2e1a4259ed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41951 } } peers { permanent_uuid: "0269a239f4624610a1ce99d87e3f8967" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44393 } } }
I20260619 09:01:33.094089 8470 tablet_replica.cc:442] T 19a175c688d54d8fae2aa105f1fca9cb P c9d6290881cd499dbaf9dd32d03583aa: This TxnStatusTablet replica's current role is: LEADER
I20260619 09:01:33.094278 7920 catalog_manager.cc:5495] T 19a175c688d54d8fae2aa105f1fca9cb P c9d6290881cd499dbaf9dd32d03583aa reported cstate change: term changed from 0 to 1, leader changed from <none> to c9d6290881cd499dbaf9dd32d03583aa (127.0.98.1). New cstate: current_term: 1 leader_uuid: "c9d6290881cd499dbaf9dd32d03583aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9d6290881cd499dbaf9dd32d03583aa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44937 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f06926dc12064dd78ea8de2e1a4259ed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41951 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0269a239f4624610a1ce99d87e3f8967" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44393 } health_report { overall_health: UNKNOWN } } }
I20260619 09:01:33.094930 8472 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260619 09:01:33.095080 8472 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260619 09:01:33.114892 8465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:33.112523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44937 (local address 127.0.0.1:48324)
0619 09:01:33.112590 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0619 09:01:33.112601 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 09:01:33.112640 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:01:33.112873 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:01:33.112874 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:01:33.112875 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 09:01:33.112927 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:33.112929 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:33.113519 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:33.113520 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:33.114148 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:33.114151 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:33.114475 (+ 324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:33.114476 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:33.114534 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:33.114549 (+ 15us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 09:01:33.114714 (+ 165us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 09:01:33.114720 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 09:01:33.114758 (+ 38us) client_negotiation.cc:240] Negotiation successful
0619 09:01:33.114808 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260619 09:01:33.114886 8428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:33.112712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48324 (local address 127.0.98.1:44937)
0619 09:01:33.112766 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:33.112768 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:33.112775 (+ 7us) server_negotiation.cc:407] Connection header received
0619 09:01:33.112798 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:33.112799 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:33.112809 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:33.112855 (+ 46us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 09:01:33.112985 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:33.113454 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:33.114227 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:33.114419 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:33.114471 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:33.114632 (+ 161us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 09:01:33.114667 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 09:01:33.114709 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0619 09:01:33.114769 (+ 60us) server_negotiation.cc:299] Negotiation successful
0619 09:01:33.114789 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260619 09:01:33.115942 8218 raft_consensus.cc:1261] T 19a175c688d54d8fae2aa105f1fca9cb P f06926dc12064dd78ea8de2e1a4259ed [term 1 FOLLOWER]: Refusing update from remote peer c9d6290881cd499dbaf9dd32d03583aa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 09:01:33.115938 8358 raft_consensus.cc:1261] T 19a175c688d54d8fae2aa105f1fca9cb P 0269a239f4624610a1ce99d87e3f8967 [term 1 FOLLOWER]: Refusing update from remote peer c9d6290881cd499dbaf9dd32d03583aa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 09:01:33.116137 8470 consensus_queue.cc:1035] T 19a175c688d54d8fae2aa105f1fca9cb P c9d6290881cd499dbaf9dd32d03583aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "0269a239f4624610a1ce99d87e3f8967" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44393 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 09:01:33.116228 8437 consensus_queue.cc:1035] T 19a175c688d54d8fae2aa105f1fca9cb P c9d6290881cd499dbaf9dd32d03583aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "f06926dc12064dd78ea8de2e1a4259ed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 09:01:33.116706 8435 tablet_replica.cc:439] T 19a175c688d54d8fae2aa105f1fca9cb P f06926dc12064dd78ea8de2e1a4259ed: TxnStatusTablet state changed. Reason: New leader c9d6290881cd499dbaf9dd32d03583aa. Latest consensus state: current_term: 1 leader_uuid: "c9d6290881cd499dbaf9dd32d03583aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9d6290881cd499dbaf9dd32d03583aa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44937 } } peers { permanent_uuid: "f06926dc12064dd78ea8de2e1a4259ed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41951 } } peers { permanent_uuid: "0269a239f4624610a1ce99d87e3f8967" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44393 } } }
I20260619 09:01:33.116740 8436 tablet_replica.cc:439] T 19a175c688d54d8fae2aa105f1fca9cb P 0269a239f4624610a1ce99d87e3f8967: TxnStatusTablet state changed. Reason: New leader c9d6290881cd499dbaf9dd32d03583aa. Latest consensus state: current_term: 1 leader_uuid: "c9d6290881cd499dbaf9dd32d03583aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9d6290881cd499dbaf9dd32d03583aa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44937 } } peers { permanent_uuid: "f06926dc12064dd78ea8de2e1a4259ed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41951 } } peers { permanent_uuid: "0269a239f4624610a1ce99d87e3f8967" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44393 } } }
I20260619 09:01:33.116791 8435 tablet_replica.cc:442] T 19a175c688d54d8fae2aa105f1fca9cb P f06926dc12064dd78ea8de2e1a4259ed: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 09:01:33.116816 8436 tablet_replica.cc:442] T 19a175c688d54d8fae2aa105f1fca9cb P 0269a239f4624610a1ce99d87e3f8967: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 09:01:33.117435 8437 tablet_replica.cc:439] T 19a175c688d54d8fae2aa105f1fca9cb P c9d6290881cd499dbaf9dd32d03583aa: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c9d6290881cd499dbaf9dd32d03583aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9d6290881cd499dbaf9dd32d03583aa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44937 } } peers { permanent_uuid: "f06926dc12064dd78ea8de2e1a4259ed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41951 } } peers { permanent_uuid: "0269a239f4624610a1ce99d87e3f8967" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44393 } } }
I20260619 09:01:33.117516 8437 tablet_replica.cc:442] T 19a175c688d54d8fae2aa105f1fca9cb P c9d6290881cd499dbaf9dd32d03583aa: This TxnStatusTablet replica's current role is: LEADER
I20260619 09:01:33.117605 8435 tablet_replica.cc:439] T 19a175c688d54d8fae2aa105f1fca9cb P f06926dc12064dd78ea8de2e1a4259ed: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c9d6290881cd499dbaf9dd32d03583aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9d6290881cd499dbaf9dd32d03583aa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44937 } } peers { permanent_uuid: "f06926dc12064dd78ea8de2e1a4259ed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41951 } } peers { permanent_uuid: "0269a239f4624610a1ce99d87e3f8967" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44393 } } }
I20260619 09:01:33.117686 8435 tablet_replica.cc:442] T 19a175c688d54d8fae2aa105f1fca9cb P f06926dc12064dd78ea8de2e1a4259ed: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 09:01:33.118110 8470 tablet_replica.cc:439] T 19a175c688d54d8fae2aa105f1fca9cb P c9d6290881cd499dbaf9dd32d03583aa: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c9d6290881cd499dbaf9dd32d03583aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9d6290881cd499dbaf9dd32d03583aa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44937 } } peers { permanent_uuid: "f06926dc12064dd78ea8de2e1a4259ed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41951 } } peers { permanent_uuid: "0269a239f4624610a1ce99d87e3f8967" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44393 } } }
I20260619 09:01:33.118186 8470 tablet_replica.cc:442] T 19a175c688d54d8fae2aa105f1fca9cb P c9d6290881cd499dbaf9dd32d03583aa: This TxnStatusTablet replica's current role is: LEADER
I20260619 09:01:33.118480 8436 tablet_replica.cc:439] T 19a175c688d54d8fae2aa105f1fca9cb P 0269a239f4624610a1ce99d87e3f8967: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c9d6290881cd499dbaf9dd32d03583aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9d6290881cd499dbaf9dd32d03583aa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44937 } } peers { permanent_uuid: "f06926dc12064dd78ea8de2e1a4259ed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41951 } } peers { permanent_uuid: "0269a239f4624610a1ce99d87e3f8967" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44393 } } }
I20260619 09:01:33.118551 8436 tablet_replica.cc:442] T 19a175c688d54d8fae2aa105f1fca9cb P 0269a239f4624610a1ce99d87e3f8967: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 09:01:33.119120 8477 mvcc.cc:205] Tried to move back new op lower bound from 7298497303001350144 to 7298497302912270336. Current Snapshot: MvccSnapshot[applied={T|T < 7298497303001350144}]
I20260619 09:01:33.121599 8485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:33.119222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38160 (local address 127.0.98.3:44393)
0619 09:01:33.119387 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:33.119389 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:33.119396 (+ 7us) server_negotiation.cc:407] Connection header received
0619 09:01:33.119458 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:33.119459 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:33.119477 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:33.119564 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 09:01:33.119685 (+ 121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:33.120197 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:33.120903 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:33.121112 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:33.121165 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:33.121300 (+ 135us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 09:01:33.121400 (+ 100us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 09:01:33.121459 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0619 09:01:33.121502 (+ 43us) server_negotiation.cc:299] Negotiation successful
0619 09:01:33.121529 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":46,"threads_started":1}
I20260619 09:01:33.126094 8428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:33.123739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54293 (local address 127.0.98.1:44937)
0619 09:01:33.123783 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:33.123785 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:33.123852 (+ 67us) server_negotiation.cc:407] Connection header received
0619 09:01:33.123919 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:33.123920 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:33.123929 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:33.123968 (+ 39us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 09:01:33.124074 (+ 106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:33.124489 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:33.125559 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:33.125727 (+ 168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:33.125767 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:33.125923 (+ 156us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 09:01:33.125946 (+ 23us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 09:01:33.125975 (+ 29us) server_negotiation.cc:1035] Waiting for connection context
0619 09:01:33.126035 (+ 60us) server_negotiation.cc:299] Negotiation successful
0619 09:01:33.126049 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260619 09:01:33.126157 8488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:33.123666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44937 (local address 127.0.98.3:54293)
0619 09:01:33.123800 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0619 09:01:33.123810 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 09:01:33.123866 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:01:33.123971 (+ 105us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:01:33.123972 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:01:33.123973 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 09:01:33.124028 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:33.124029 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:33.124558 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:33.124559 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:33.125493 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:33.125496 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:33.125782 (+ 286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:33.125783 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:33.125844 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:33.125857 (+ 13us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 09:01:33.125989 (+ 132us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 09:01:33.125990 (+ 1us) client_negotiation.cc:769] Sending connection context
0619 09:01:33.126033 (+ 43us) client_negotiation.cc:240] Negotiation successful
0619 09:01:33.126075 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":67,"threads_started":1}
I20260619 09:01:33.128021 8218 raft_consensus.cc:1261] T 86b6850e7c494e38985d8e3b55df01e5 P f06926dc12064dd78ea8de2e1a4259ed [term 1 FOLLOWER]: Refusing update from remote peer 0269a239f4624610a1ce99d87e3f8967: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 09:01:33.128065 8078 raft_consensus.cc:1261] T 86b6850e7c494e38985d8e3b55df01e5 P c9d6290881cd499dbaf9dd32d03583aa [term 1 FOLLOWER]: Refusing update from remote peer 0269a239f4624610a1ce99d87e3f8967: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 09:01:33.128276 8458 consensus_queue.cc:1035] T 86b6850e7c494e38985d8e3b55df01e5 P 0269a239f4624610a1ce99d87e3f8967 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9d6290881cd499dbaf9dd32d03583aa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 09:01:33.128440 8436 consensus_queue.cc:1035] T 86b6850e7c494e38985d8e3b55df01e5 P 0269a239f4624610a1ce99d87e3f8967 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f06926dc12064dd78ea8de2e1a4259ed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 09:01:33.137545 8485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:33.133373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60301 (local address 127.0.98.3:44393)
0619 09:01:33.133407 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:33.133408 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:33.134987 (+ 1579us) server_negotiation.cc:407] Connection header received
0619 09:01:33.135013 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:33.135014 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:33.135025 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:33.135088 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 09:01:33.135248 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:33.135989 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:33.136842 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:33.137138 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:33.137203 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:33.137333 (+ 130us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 09:01:33.137363 (+ 30us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 09:01:33.137415 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0619 09:01:33.137456 (+ 41us) server_negotiation.cc:299] Negotiation successful
0619 09:01:33.137474 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260619 09:01:33.137554 8495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:33.133330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44393 (local address 127.0.98.1:60301)
0619 09:01:33.134921 (+ 1591us) negotiation.cc:107] Waiting for socket to connect
0619 09:01:33.134930 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 09:01:33.134981 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:01:33.135105 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:01:33.135106 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:01:33.135109 (+ 3us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 09:01:33.135165 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:33.135167 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:33.136068 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:33.136069 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:33.136774 (+ 705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:33.136777 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:33.137205 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:33.137206 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:33.137256 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:33.137268 (+ 12us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 09:01:33.137423 (+ 155us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 09:01:33.137424 (+ 1us) client_negotiation.cc:769] Sending connection context
0619 09:01:33.137451 (+ 27us) client_negotiation.cc:240] Negotiation successful
0619 09:01:33.137483 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1553,"thread_start_us":70,"threads_started":1}
I20260619 09:01:33.147159 7920 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33932:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 09:01:33.147260 7920 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33932:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 09:01:33.147852 7920 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 75e0d761165f4e3f9aa72ac34214c70c: Sending DeleteTablet for 3 replicas of tablet 86b6850e7c494e38985d8e3b55df01e5
I20260619 09:01:33.148149 8058 tablet_service.cc:1514] Processing DeleteTablet for tablet 86b6850e7c494e38985d8e3b55df01e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 09:01:33 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48302
I20260619 09:01:33.148427 8198 tablet_service.cc:1514] Processing DeleteTablet for tablet 86b6850e7c494e38985d8e3b55df01e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 09:01:33 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51328
I20260619 09:01:33.148697 8338 tablet_service.cc:1514] Processing DeleteTablet for tablet 86b6850e7c494e38985d8e3b55df01e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 09:01:33 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38158
I20260619 09:01:33.154907 8498 tablet_replica.cc:330] T 86b6850e7c494e38985d8e3b55df01e5 P c9d6290881cd499dbaf9dd32d03583aa: stopping tablet replica
I20260619 09:01:33.154927 8499 tablet_replica.cc:330] T 86b6850e7c494e38985d8e3b55df01e5 P f06926dc12064dd78ea8de2e1a4259ed: stopping tablet replica
I20260619 09:01:33.154980 8498 raft_consensus.cc:2229] T 86b6850e7c494e38985d8e3b55df01e5 P c9d6290881cd499dbaf9dd32d03583aa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 09:01:33.154980 8499 raft_consensus.cc:2229] T 86b6850e7c494e38985d8e3b55df01e5 P f06926dc12064dd78ea8de2e1a4259ed [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 09:01:33.155016 8498 raft_consensus.cc:2258] T 86b6850e7c494e38985d8e3b55df01e5 P c9d6290881cd499dbaf9dd32d03583aa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 09:01:33.155016 8499 raft_consensus.cc:2258] T 86b6850e7c494e38985d8e3b55df01e5 P f06926dc12064dd78ea8de2e1a4259ed [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 09:01:33.155263 8498 ts_tablet_manager.cc:1905] T 86b6850e7c494e38985d8e3b55df01e5 P c9d6290881cd499dbaf9dd32d03583aa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 09:01:33.155263 8499 ts_tablet_manager.cc:1905] T 86b6850e7c494e38985d8e3b55df01e5 P f06926dc12064dd78ea8de2e1a4259ed: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 09:01:33.156831 8500 tablet_replica.cc:330] T 86b6850e7c494e38985d8e3b55df01e5 P 0269a239f4624610a1ce99d87e3f8967: stopping tablet replica
I20260619 09:01:33.156898 8500 raft_consensus.cc:2229] T 86b6850e7c494e38985d8e3b55df01e5 P 0269a239f4624610a1ce99d87e3f8967 [term 1 LEADER]: Raft consensus shutting down.
I20260619 09:01:33.156986 8500 raft_consensus.cc:2258] T 86b6850e7c494e38985d8e3b55df01e5 P 0269a239f4624610a1ce99d87e3f8967 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 09:01:33.157107 8499 ts_tablet_manager.cc:1918] T 86b6850e7c494e38985d8e3b55df01e5 P f06926dc12064dd78ea8de2e1a4259ed: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 09:01:33.157162 8499 log.cc:1196] T 86b6850e7c494e38985d8e3b55df01e5 P f06926dc12064dd78ea8de2e1a4259ed: Deleting WAL directory at /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781859658134510-392-0/minicluster-data/ts-1/wal/wals/86b6850e7c494e38985d8e3b55df01e5
I20260619 09:01:33.157539 8499 ts_tablet_manager.cc:1939] T 86b6850e7c494e38985d8e3b55df01e5 P f06926dc12064dd78ea8de2e1a4259ed: Deleting consensus metadata
I20260619 09:01:33.157830 7907 catalog_manager.cc:4841] TS f06926dc12064dd78ea8de2e1a4259ed (127.0.98.2:41951): tablet 86b6850e7c494e38985d8e3b55df01e5 (table test-table [id=ed9e1d37b96b4dcf9df55d4034f62081]) successfully deleted
I20260619 09:01:33.157946 8498 ts_tablet_manager.cc:1918] T 86b6850e7c494e38985d8e3b55df01e5 P c9d6290881cd499dbaf9dd32d03583aa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 09:01:33.157999 8498 log.cc:1196] T 86b6850e7c494e38985d8e3b55df01e5 P c9d6290881cd499dbaf9dd32d03583aa: Deleting WAL directory at /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781859658134510-392-0/minicluster-data/ts-0/wal/wals/86b6850e7c494e38985d8e3b55df01e5
I20260619 09:01:33.158123 8500 ts_tablet_manager.cc:1905] T 86b6850e7c494e38985d8e3b55df01e5 P 0269a239f4624610a1ce99d87e3f8967: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 09:01:33.158178 8498 ts_tablet_manager.cc:1939] T 86b6850e7c494e38985d8e3b55df01e5 P c9d6290881cd499dbaf9dd32d03583aa: Deleting consensus metadata
I20260619 09:01:33.158435 7907 catalog_manager.cc:4841] TS c9d6290881cd499dbaf9dd32d03583aa (127.0.98.1:44937): tablet 86b6850e7c494e38985d8e3b55df01e5 (table test-table [id=ed9e1d37b96b4dcf9df55d4034f62081]) successfully deleted
I20260619 09:01:33.159713 8500 ts_tablet_manager.cc:1918] T 86b6850e7c494e38985d8e3b55df01e5 P 0269a239f4624610a1ce99d87e3f8967: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 09:01:33.159771 8500 log.cc:1196] T 86b6850e7c494e38985d8e3b55df01e5 P 0269a239f4624610a1ce99d87e3f8967: Deleting WAL directory at /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781859658134510-392-0/minicluster-data/ts-2/wal/wals/86b6850e7c494e38985d8e3b55df01e5
I20260619 09:01:33.159967 8500 ts_tablet_manager.cc:1939] T 86b6850e7c494e38985d8e3b55df01e5 P 0269a239f4624610a1ce99d87e3f8967: Deleting consensus metadata
I20260619 09:01:33.160287 7907 catalog_manager.cc:4841] TS 0269a239f4624610a1ce99d87e3f8967 (127.0.98.3:44393): tablet 86b6850e7c494e38985d8e3b55df01e5 (table test-table [id=ed9e1d37b96b4dcf9df55d4034f62081]) successfully deleted
Jun 19 09:01:33 dist-test-slave-mtdr krb5kdc[7876](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859693, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 09:01:33 dist-test-slave-mtdr krb5kdc[7876](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 09:01:33 dist-test-slave-mtdr krb5kdc[7876](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260619 09:01:33.180138 8466 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 09:01:33.173324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33954 (local address 127.0.98.62:35301)
0619 09:01:33.173369 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:33.173371 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:33.173560 (+ 189us) server_negotiation.cc:407] Connection header received
0619 09:01:33.173634 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:33.173636 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:33.173648 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:33.173707 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:01:33.174375 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:33.175282 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:33.176422 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:33.176744 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:33.176827 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:33.180072 (+ 3245us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33954: BlockingRecv error: recv got EOF from 127.0.0.1:33954 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
Jun 19 09:01:33 dist-test-slave-mtdr krb5kdc[7876](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859693, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260619 09:01:33.189194 8466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:33.180810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33968 (local address 127.0.98.62:35301)
0619 09:01:33.180845 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:33.180847 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:33.181045 (+ 198us) server_negotiation.cc:407] Connection header received
0619 09:01:33.181134 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:33.181135 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:33.181145 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:33.181171 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:01:33.181759 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:33.182508 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:33.183511 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:33.183808 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:33.183875 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:33.186345 (+ 2470us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:01:33.186347 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:01:33.186347 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:01:33.186354 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:01:33.188042 (+ 1688us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:01:33.188432 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:01:33.188433 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:01:33.188433 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:01:33.188497 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:01:33.188685 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:01:33.188686 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:01:33.188687 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:01:33.188808 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:01:33.188897 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0619 09:01:33.189020 (+ 123us) server_negotiation.cc:299] Negotiation successful
0619 09:01:33.189097 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260619 09:01:33.190693 7920 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33968:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 09:01:33.190904 7920 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 09:01:33.192217 8058 tablet_service.cc:1467] Processing CreateTablet for tablet 3d44db37d9d541c3957bf34c3c79435f (DEFAULT_TABLE table=test-table [id=7f2c7e19489c4ce4b3e4cde1089e72ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 09:01:33.192345 8058 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d44db37d9d541c3957bf34c3c79435f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 09:01:33.192340 8198 tablet_service.cc:1467] Processing CreateTablet for tablet 3d44db37d9d541c3957bf34c3c79435f (DEFAULT_TABLE table=test-table [id=7f2c7e19489c4ce4b3e4cde1089e72ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 09:01:33.192462 8198 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d44db37d9d541c3957bf34c3c79435f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 09:01:33.193707 8429 tablet_bootstrap.cc:492] T 3d44db37d9d541c3957bf34c3c79435f P c9d6290881cd499dbaf9dd32d03583aa: Bootstrap starting.
I20260619 09:01:33.193800 8338 tablet_service.cc:1467] Processing CreateTablet for tablet 3d44db37d9d541c3957bf34c3c79435f (DEFAULT_TABLE table=test-table [id=7f2c7e19489c4ce4b3e4cde1089e72ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 09:01:33.193904 8338 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d44db37d9d541c3957bf34c3c79435f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 09:01:33.194037 8429 tablet_bootstrap.cc:654] T 3d44db37d9d541c3957bf34c3c79435f P c9d6290881cd499dbaf9dd32d03583aa: Neither blocks nor log segments found. Creating new log.
I20260619 09:01:33.194352 8431 tablet_bootstrap.cc:492] T 3d44db37d9d541c3957bf34c3c79435f P f06926dc12064dd78ea8de2e1a4259ed: Bootstrap starting.
I20260619 09:01:33.194736 8431 tablet_bootstrap.cc:654] T 3d44db37d9d541c3957bf34c3c79435f P f06926dc12064dd78ea8de2e1a4259ed: Neither blocks nor log segments found. Creating new log.
I20260619 09:01:33.195114 8430 tablet_bootstrap.cc:492] T 3d44db37d9d541c3957bf34c3c79435f P 0269a239f4624610a1ce99d87e3f8967: Bootstrap starting.
I20260619 09:01:33.195436 8429 tablet_bootstrap.cc:492] T 3d44db37d9d541c3957bf34c3c79435f P c9d6290881cd499dbaf9dd32d03583aa: No bootstrap required, opened a new log
I20260619 09:01:33.195492 8429 ts_tablet_manager.cc:1397] T 3d44db37d9d541c3957bf34c3c79435f P c9d6290881cd499dbaf9dd32d03583aa: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 09:01:33.195535 8430 tablet_bootstrap.cc:654] T 3d44db37d9d541c3957bf34c3c79435f P 0269a239f4624610a1ce99d87e3f8967: Neither blocks nor log segments found. Creating new log.
I20260619 09:01:33.195653 8429 raft_consensus.cc:348] T 3d44db37d9d541c3957bf34c3c79435f P c9d6290881cd499dbaf9dd32d03583aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9d6290881cd499dbaf9dd32d03583aa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44937 } } peers { permanent_uuid: "f06926dc12064dd78ea8de2e1a4259ed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41951 } } peers { permanent_uuid: "0269a239f4624610a1ce99d87e3f8967" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44393 } }
I20260619 09:01:33.195716 8429 raft_consensus.cc:374] T 3d44db37d9d541c3957bf34c3c79435f P c9d6290881cd499dbaf9dd32d03583aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 09:01:33.195735 8429 raft_consensus.cc:729] T 3d44db37d9d541c3957bf34c3c79435f P c9d6290881cd499dbaf9dd32d03583aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9d6290881cd499dbaf9dd32d03583aa, State: Initialized, Role: FOLLOWER
I20260619 09:01:33.195775 8429 consensus_queue.cc:260] T 3d44db37d9d541c3957bf34c3c79435f P c9d6290881cd499dbaf9dd32d03583aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9d6290881cd499dbaf9dd32d03583aa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44937 } } peers { permanent_uuid: "f06926dc12064dd78ea8de2e1a4259ed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41951 } } peers { permanent_uuid: "0269a239f4624610a1ce99d87e3f8967" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44393 } }
I20260619 09:01:33.195852 8429 ts_tablet_manager.cc:1428] T 3d44db37d9d541c3957bf34c3c79435f P c9d6290881cd499dbaf9dd32d03583aa: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260619 09:01:33.196848 8430 tablet_bootstrap.cc:492] T 3d44db37d9d541c3957bf34c3c79435f P 0269a239f4624610a1ce99d87e3f8967: No bootstrap required, opened a new log
I20260619 09:01:33.196892 8430 ts_tablet_manager.cc:1397] T 3d44db37d9d541c3957bf34c3c79435f P 0269a239f4624610a1ce99d87e3f8967: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260619 09:01:33.197009 8430 raft_consensus.cc:348] T 3d44db37d9d541c3957bf34c3c79435f P 0269a239f4624610a1ce99d87e3f8967 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9d6290881cd499dbaf9dd32d03583aa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44937 } } peers { permanent_uuid: "f06926dc12064dd78ea8de2e1a4259ed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41951 } } peers { permanent_uuid: "0269a239f4624610a1ce99d87e3f8967" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44393 } }
I20260619 09:01:33.197069 8430 raft_consensus.cc:374] T 3d44db37d9d541c3957bf34c3c79435f P 0269a239f4624610a1ce99d87e3f8967 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 09:01:33.197088 8430 raft_consensus.cc:729] T 3d44db37d9d541c3957bf34c3c79435f P 0269a239f4624610a1ce99d87e3f8967 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0269a239f4624610a1ce99d87e3f8967, State: Initialized, Role: FOLLOWER
I20260619 09:01:33.197124 8430 consensus_queue.cc:260] T 3d44db37d9d541c3957bf34c3c79435f P 0269a239f4624610a1ce99d87e3f8967 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9d6290881cd499dbaf9dd32d03583aa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44937 } } peers { permanent_uuid: "f06926dc12064dd78ea8de2e1a4259ed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41951 } } peers { permanent_uuid: "0269a239f4624610a1ce99d87e3f8967" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44393 } }
I20260619 09:01:33.197194 8430 ts_tablet_manager.cc:1428] T 3d44db37d9d541c3957bf34c3c79435f P 0269a239f4624610a1ce99d87e3f8967: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260619 09:01:33.197816 8431 tablet_bootstrap.cc:492] T 3d44db37d9d541c3957bf34c3c79435f P f06926dc12064dd78ea8de2e1a4259ed: No bootstrap required, opened a new log
I20260619 09:01:33.197870 8431 ts_tablet_manager.cc:1397] T 3d44db37d9d541c3957bf34c3c79435f P f06926dc12064dd78ea8de2e1a4259ed: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.001s
I20260619 09:01:33.197997 8431 raft_consensus.cc:348] T 3d44db37d9d541c3957bf34c3c79435f P f06926dc12064dd78ea8de2e1a4259ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9d6290881cd499dbaf9dd32d03583aa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44937 } } peers { permanent_uuid: "f06926dc12064dd78ea8de2e1a4259ed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41951 } } peers { permanent_uuid: "0269a239f4624610a1ce99d87e3f8967" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44393 } }
I20260619 09:01:33.198066 8431 raft_consensus.cc:374] T 3d44db37d9d541c3957bf34c3c79435f P f06926dc12064dd78ea8de2e1a4259ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 09:01:33.198086 8431 raft_consensus.cc:729] T 3d44db37d9d541c3957bf34c3c79435f P f06926dc12064dd78ea8de2e1a4259ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f06926dc12064dd78ea8de2e1a4259ed, State: Initialized, Role: FOLLOWER
I20260619 09:01:33.198129 8431 consensus_queue.cc:260] T 3d44db37d9d541c3957bf34c3c79435f P f06926dc12064dd78ea8de2e1a4259ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9d6290881cd499dbaf9dd32d03583aa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44937 } } peers { permanent_uuid: "f06926dc12064dd78ea8de2e1a4259ed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41951 } } peers { permanent_uuid: "0269a239f4624610a1ce99d87e3f8967" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44393 } }
I20260619 09:01:33.198207 8431 ts_tablet_manager.cc:1428] T 3d44db37d9d541c3957bf34c3c79435f P f06926dc12064dd78ea8de2e1a4259ed: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260619 09:01:33.229192 8437 raft_consensus.cc:482] T 3d44db37d9d541c3957bf34c3c79435f P c9d6290881cd499dbaf9dd32d03583aa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 09:01:33.229279 8437 raft_consensus.cc:504] T 3d44db37d9d541c3957bf34c3c79435f P c9d6290881cd499dbaf9dd32d03583aa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9d6290881cd499dbaf9dd32d03583aa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44937 } } peers { permanent_uuid: "f06926dc12064dd78ea8de2e1a4259ed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41951 } } peers { permanent_uuid: "0269a239f4624610a1ce99d87e3f8967" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44393 } }
I20260619 09:01:33.229422 8437 leader_election.cc:288] T 3d44db37d9d541c3957bf34c3c79435f P c9d6290881cd499dbaf9dd32d03583aa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f06926dc12064dd78ea8de2e1a4259ed (127.0.98.2:41951), 0269a239f4624610a1ce99d87e3f8967 (127.0.98.3:44393)
I20260619 09:01:33.229627 8218 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d44db37d9d541c3957bf34c3c79435f" candidate_uuid: "c9d6290881cd499dbaf9dd32d03583aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f06926dc12064dd78ea8de2e1a4259ed" is_pre_election: true
I20260619 09:01:33.229629 8358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d44db37d9d541c3957bf34c3c79435f" candidate_uuid: "c9d6290881cd499dbaf9dd32d03583aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0269a239f4624610a1ce99d87e3f8967" is_pre_election: true
I20260619 09:01:33.229719 8358 raft_consensus.cc:2454] T 3d44db37d9d541c3957bf34c3c79435f P 0269a239f4624610a1ce99d87e3f8967 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c9d6290881cd499dbaf9dd32d03583aa in term 0.
I20260619 09:01:33.229719 8218 raft_consensus.cc:2454] T 3d44db37d9d541c3957bf34c3c79435f P f06926dc12064dd78ea8de2e1a4259ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c9d6290881cd499dbaf9dd32d03583aa in term 0.
I20260619 09:01:33.229849 8011 leader_election.cc:302] T 3d44db37d9d541c3957bf34c3c79435f P c9d6290881cd499dbaf9dd32d03583aa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0269a239f4624610a1ce99d87e3f8967, c9d6290881cd499dbaf9dd32d03583aa; no voters:
I20260619 09:01:33.229940 8437 raft_consensus.cc:2781] T 3d44db37d9d541c3957bf34c3c79435f P c9d6290881cd499dbaf9dd32d03583aa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 09:01:33.229980 8437 raft_consensus.cc:482] T 3d44db37d9d541c3957bf34c3c79435f P c9d6290881cd499dbaf9dd32d03583aa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 09:01:33.230000 8437 raft_consensus.cc:3037] T 3d44db37d9d541c3957bf34c3c79435f P c9d6290881cd499dbaf9dd32d03583aa [term 0 FOLLOWER]: Advancing to term 1
I20260619 09:01:33.230597 8437 raft_consensus.cc:504] T 3d44db37d9d541c3957bf34c3c79435f P c9d6290881cd499dbaf9dd32d03583aa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9d6290881cd499dbaf9dd32d03583aa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44937 } } peers { permanent_uuid: "f06926dc12064dd78ea8de2e1a4259ed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41951 } } peers { permanent_uuid: "0269a239f4624610a1ce99d87e3f8967" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44393 } }
I20260619 09:01:33.230707 8437 leader_election.cc:288] T 3d44db37d9d541c3957bf34c3c79435f P c9d6290881cd499dbaf9dd32d03583aa [CANDIDATE]: Term 1 election: Requested vote from peers f06926dc12064dd78ea8de2e1a4259ed (127.0.98.2:41951), 0269a239f4624610a1ce99d87e3f8967 (127.0.98.3:44393)
I20260619 09:01:33.230894 8218 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d44db37d9d541c3957bf34c3c79435f" candidate_uuid: "c9d6290881cd499dbaf9dd32d03583aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f06926dc12064dd78ea8de2e1a4259ed"
I20260619 09:01:33.230964 8218 raft_consensus.cc:3037] T 3d44db37d9d541c3957bf34c3c79435f P f06926dc12064dd78ea8de2e1a4259ed [term 0 FOLLOWER]: Advancing to term 1
I20260619 09:01:33.230959 8358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d44db37d9d541c3957bf34c3c79435f" candidate_uuid: "c9d6290881cd499dbaf9dd32d03583aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0269a239f4624610a1ce99d87e3f8967"
I20260619 09:01:33.231063 8358 raft_consensus.cc:3037] T 3d44db37d9d541c3957bf34c3c79435f P 0269a239f4624610a1ce99d87e3f8967 [term 0 FOLLOWER]: Advancing to term 1
I20260619 09:01:33.231555 8218 raft_consensus.cc:2454] T 3d44db37d9d541c3957bf34c3c79435f P f06926dc12064dd78ea8de2e1a4259ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9d6290881cd499dbaf9dd32d03583aa in term 1.
I20260619 09:01:33.231702 8011 leader_election.cc:302] T 3d44db37d9d541c3957bf34c3c79435f P c9d6290881cd499dbaf9dd32d03583aa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c9d6290881cd499dbaf9dd32d03583aa, f06926dc12064dd78ea8de2e1a4259ed; no voters:
I20260619 09:01:33.231797 8437 raft_consensus.cc:2781] T 3d44db37d9d541c3957bf34c3c79435f P c9d6290881cd499dbaf9dd32d03583aa [term 1 FOLLOWER]: Leader election won for term 1
I20260619 09:01:33.231834 8437 raft_consensus.cc:686] T 3d44db37d9d541c3957bf34c3c79435f P c9d6290881cd499dbaf9dd32d03583aa [term 1 LEADER]: Becoming Leader. State: Replica: c9d6290881cd499dbaf9dd32d03583aa, State: Running, Role: LEADER
I20260619 09:01:33.231848 8358 raft_consensus.cc:2454] T 3d44db37d9d541c3957bf34c3c79435f P 0269a239f4624610a1ce99d87e3f8967 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9d6290881cd499dbaf9dd32d03583aa in term 1.
I20260619 09:01:33.231876 8437 consensus_queue.cc:237] T 3d44db37d9d541c3957bf34c3c79435f P c9d6290881cd499dbaf9dd32d03583aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9d6290881cd499dbaf9dd32d03583aa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44937 } } peers { permanent_uuid: "f06926dc12064dd78ea8de2e1a4259ed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41951 } } peers { permanent_uuid: "0269a239f4624610a1ce99d87e3f8967" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44393 } }
I20260619 09:01:33.232460 7919 catalog_manager.cc:5495] T 3d44db37d9d541c3957bf34c3c79435f P c9d6290881cd499dbaf9dd32d03583aa reported cstate change: term changed from 0 to 1, leader changed from <none> to c9d6290881cd499dbaf9dd32d03583aa (127.0.98.1). New cstate: current_term: 1 leader_uuid: "c9d6290881cd499dbaf9dd32d03583aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9d6290881cd499dbaf9dd32d03583aa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44937 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f06926dc12064dd78ea8de2e1a4259ed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41951 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0269a239f4624610a1ce99d87e3f8967" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44393 } health_report { overall_health: UNKNOWN } } }
I20260619 09:01:33.236683 8428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:33.234489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48334 (local address 127.0.98.1:44937)
0619 09:01:33.234545 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:33.234547 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:33.234554 (+ 7us) server_negotiation.cc:407] Connection header received
0619 09:01:33.234577 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:33.234578 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:33.234587 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:33.234631 (+ 44us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 09:01:33.234725 (+ 94us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:33.235242 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:33.236089 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:33.236291 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:33.236346 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:33.236472 (+ 126us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 09:01:33.236506 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 09:01:33.236546 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0619 09:01:33.236601 (+ 55us) server_negotiation.cc:299] Negotiation successful
0619 09:01:33.236619 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260619 09:01:33.237458 8358 raft_consensus.cc:1261] T 3d44db37d9d541c3957bf34c3c79435f P 0269a239f4624610a1ce99d87e3f8967 [term 1 FOLLOWER]: Refusing update from remote peer c9d6290881cd499dbaf9dd32d03583aa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 09:01:33.237468 8218 raft_consensus.cc:1261] T 3d44db37d9d541c3957bf34c3c79435f P f06926dc12064dd78ea8de2e1a4259ed [term 1 FOLLOWER]: Refusing update from remote peer c9d6290881cd499dbaf9dd32d03583aa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 09:01:33.237679 8437 consensus_queue.cc:1035] T 3d44db37d9d541c3957bf34c3c79435f P c9d6290881cd499dbaf9dd32d03583aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "f06926dc12064dd78ea8de2e1a4259ed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 09:01:33.237800 8470 consensus_queue.cc:1035] T 3d44db37d9d541c3957bf34c3c79435f P c9d6290881cd499dbaf9dd32d03583aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "0269a239f4624610a1ce99d87e3f8967" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44393 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 09:01:33.240173 7919 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33968:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 09:01:33.240258 7919 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33968:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 09:01:33.240839 7919 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 75e0d761165f4e3f9aa72ac34214c70c: Sending DeleteTablet for 3 replicas of tablet 3d44db37d9d541c3957bf34c3c79435f
I20260619 09:01:33.241180 8198 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d44db37d9d541c3957bf34c3c79435f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 09:01:33 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51328
I20260619 09:01:33.241271 8338 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d44db37d9d541c3957bf34c3c79435f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 09:01:33 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38158
I20260619 09:01:33.241354 8499 tablet_replica.cc:330] T 3d44db37d9d541c3957bf34c3c79435f P f06926dc12064dd78ea8de2e1a4259ed: stopping tablet replica
I20260619 09:01:33.241408 8499 raft_consensus.cc:2229] T 3d44db37d9d541c3957bf34c3c79435f P f06926dc12064dd78ea8de2e1a4259ed [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 09:01:33.241449 8499 raft_consensus.cc:2258] T 3d44db37d9d541c3957bf34c3c79435f P f06926dc12064dd78ea8de2e1a4259ed [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 09:01:33.241520 8500 tablet_replica.cc:330] T 3d44db37d9d541c3957bf34c3c79435f P 0269a239f4624610a1ce99d87e3f8967: stopping tablet replica
I20260619 09:01:33.241570 8500 raft_consensus.cc:2229] T 3d44db37d9d541c3957bf34c3c79435f P 0269a239f4624610a1ce99d87e3f8967 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 09:01:33.241616 8500 raft_consensus.cc:2258] T 3d44db37d9d541c3957bf34c3c79435f P 0269a239f4624610a1ce99d87e3f8967 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 09:01:33.241874 8058 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d44db37d9d541c3957bf34c3c79435f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 09:01:33 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48302
I20260619 09:01:33.241954 8498 tablet_replica.cc:330] T 3d44db37d9d541c3957bf34c3c79435f P c9d6290881cd499dbaf9dd32d03583aa: stopping tablet replica
I20260619 09:01:33.242003 8498 raft_consensus.cc:2229] T 3d44db37d9d541c3957bf34c3c79435f P c9d6290881cd499dbaf9dd32d03583aa [term 1 LEADER]: Raft consensus shutting down.
I20260619 09:01:33.242054 8498 raft_consensus.cc:2258] T 3d44db37d9d541c3957bf34c3c79435f P c9d6290881cd499dbaf9dd32d03583aa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 09:01:33.242432 7920 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33968:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 09:01:33.242587 7920 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 09:01:33.242626 8498 ts_tablet_manager.cc:1905] T 3d44db37d9d541c3957bf34c3c79435f P c9d6290881cd499dbaf9dd32d03583aa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 09:01:33.244128 8498 ts_tablet_manager.cc:1918] T 3d44db37d9d541c3957bf34c3c79435f P c9d6290881cd499dbaf9dd32d03583aa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 09:01:33.244191 8498 log.cc:1196] T 3d44db37d9d541c3957bf34c3c79435f P c9d6290881cd499dbaf9dd32d03583aa: Deleting WAL directory at /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781859658134510-392-0/minicluster-data/ts-0/wal/wals/3d44db37d9d541c3957bf34c3c79435f
I20260619 09:01:33.244374 8498 ts_tablet_manager.cc:1939] T 3d44db37d9d541c3957bf34c3c79435f P c9d6290881cd499dbaf9dd32d03583aa: Deleting consensus metadata
I20260619 09:01:33.244796 7907 catalog_manager.cc:4841] TS c9d6290881cd499dbaf9dd32d03583aa (127.0.98.1:44937): tablet 3d44db37d9d541c3957bf34c3c79435f (table test-table [id=7f2c7e19489c4ce4b3e4cde1089e72ec]) successfully deleted
I20260619 09:01:33.244870 8058 tablet_service.cc:1467] Processing CreateTablet for tablet 89f39eeaacbc4a9497acf8423b97e6b4 (DEFAULT_TABLE table=test-table [id=873634747b5448429c2aa16f5f44deeb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 09:01:33.244971 8058 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89f39eeaacbc4a9497acf8423b97e6b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 09:01:33.245047 8198 tablet_service.cc:1467] Processing CreateTablet for tablet 89f39eeaacbc4a9497acf8423b97e6b4 (DEFAULT_TABLE table=test-table [id=873634747b5448429c2aa16f5f44deeb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 09:01:33.245072 8338 tablet_service.cc:1467] Processing CreateTablet for tablet 89f39eeaacbc4a9497acf8423b97e6b4 (DEFAULT_TABLE table=test-table [id=873634747b5448429c2aa16f5f44deeb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 09:01:33.245132 8198 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89f39eeaacbc4a9497acf8423b97e6b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 09:01:33.245179 8338 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89f39eeaacbc4a9497acf8423b97e6b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 09:01:33.246392 8429 tablet_bootstrap.cc:492] T 89f39eeaacbc4a9497acf8423b97e6b4 P c9d6290881cd499dbaf9dd32d03583aa: Bootstrap starting.
I20260619 09:01:33.246481 8431 tablet_bootstrap.cc:492] T 89f39eeaacbc4a9497acf8423b97e6b4 P f06926dc12064dd78ea8de2e1a4259ed: Bootstrap starting.
I20260619 09:01:33.246511 8430 tablet_bootstrap.cc:492] T 89f39eeaacbc4a9497acf8423b97e6b4 P 0269a239f4624610a1ce99d87e3f8967: Bootstrap starting.
I20260619 09:01:33.246688 8429 tablet_bootstrap.cc:654] T 89f39eeaacbc4a9497acf8423b97e6b4 P c9d6290881cd499dbaf9dd32d03583aa: Neither blocks nor log segments found. Creating new log.
I20260619 09:01:33.246855 8431 tablet_bootstrap.cc:654] T 89f39eeaacbc4a9497acf8423b97e6b4 P f06926dc12064dd78ea8de2e1a4259ed: Neither blocks nor log segments found. Creating new log.
I20260619 09:01:33.246901 8430 tablet_bootstrap.cc:654] T 89f39eeaacbc4a9497acf8423b97e6b4 P 0269a239f4624610a1ce99d87e3f8967: Neither blocks nor log segments found. Creating new log.
I20260619 09:01:33.247212 8500 ts_tablet_manager.cc:1905] T 3d44db37d9d541c3957bf34c3c79435f P 0269a239f4624610a1ce99d87e3f8967: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 09:01:33.247637 8430 tablet_bootstrap.cc:492] T 89f39eeaacbc4a9497acf8423b97e6b4 P 0269a239f4624610a1ce99d87e3f8967: No bootstrap required, opened a new log
I20260619 09:01:33.247684 8430 ts_tablet_manager.cc:1397] T 89f39eeaacbc4a9497acf8423b97e6b4 P 0269a239f4624610a1ce99d87e3f8967: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260619 09:01:33.247788 8431 tablet_bootstrap.cc:492] T 89f39eeaacbc4a9497acf8423b97e6b4 P f06926dc12064dd78ea8de2e1a4259ed: No bootstrap required, opened a new log
I20260619 09:01:33.247821 8431 ts_tablet_manager.cc:1397] T 89f39eeaacbc4a9497acf8423b97e6b4 P f06926dc12064dd78ea8de2e1a4259ed: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260619 09:01:33.247831 8430 raft_consensus.cc:348] T 89f39eeaacbc4a9497acf8423b97e6b4 P 0269a239f4624610a1ce99d87e3f8967 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9d6290881cd499dbaf9dd32d03583aa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44937 } } peers { permanent_uuid: "f06926dc12064dd78ea8de2e1a4259ed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41951 } } peers { permanent_uuid: "0269a239f4624610a1ce99d87e3f8967" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44393 } }
I20260619 09:01:33.247896 8430 raft_consensus.cc:374] T 89f39eeaacbc4a9497acf8423b97e6b4 P 0269a239f4624610a1ce99d87e3f8967 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 09:01:33.247913 8430 raft_consensus.cc:729] T 89f39eeaacbc4a9497acf8423b97e6b4 P 0269a239f4624610a1ce99d87e3f8967 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0269a239f4624610a1ce99d87e3f8967, State: Initialized, Role: FOLLOWER
I20260619 09:01:33.247952 8430 consensus_queue.cc:260] T 89f39eeaacbc4a9497acf8423b97e6b4 P 0269a239f4624610a1ce99d87e3f8967 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9d6290881cd499dbaf9dd32d03583aa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44937 } } peers { permanent_uuid: "f06926dc12064dd78ea8de2e1a4259ed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41951 } } peers { permanent_uuid: "0269a239f4624610a1ce99d87e3f8967" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44393 } }
I20260619 09:01:33.247941 8431 raft_consensus.cc:348] T 89f39eeaacbc4a9497acf8423b97e6b4 P f06926dc12064dd78ea8de2e1a4259ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9d6290881cd499dbaf9dd32d03583aa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44937 } } peers { permanent_uuid: "f06926dc12064dd78ea8de2e1a4259ed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41951 } } peers { permanent_uuid: "0269a239f4624610a1ce99d87e3f8967" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44393 } }
I20260619 09:01:33.247995 8431 raft_consensus.cc:374] T 89f39eeaacbc4a9497acf8423b97e6b4 P f06926dc12064dd78ea8de2e1a4259ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 09:01:33.248013 8431 raft_consensus.cc:729] T 89f39eeaacbc4a9497acf8423b97e6b4 P f06926dc12064dd78ea8de2e1a4259ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f06926dc12064dd78ea8de2e1a4259ed, State: Initialized, Role: FOLLOWER
I20260619 09:01:33.248049 8430 ts_tablet_manager.cc:1428] T 89f39eeaacbc4a9497acf8423b97e6b4 P 0269a239f4624610a1ce99d87e3f8967: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260619 09:01:33.248049 8431 consensus_queue.cc:260] T 89f39eeaacbc4a9497acf8423b97e6b4 P f06926dc12064dd78ea8de2e1a4259ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9d6290881cd499dbaf9dd32d03583aa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44937 } } peers { permanent_uuid: "f06926dc12064dd78ea8de2e1a4259ed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41951 } } peers { permanent_uuid: "0269a239f4624610a1ce99d87e3f8967" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44393 } }
I20260619 09:01:33.248127 8431 ts_tablet_manager.cc:1428] T 89f39eeaacbc4a9497acf8423b97e6b4 P f06926dc12064dd78ea8de2e1a4259ed: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260619 09:01:33.248476 8500 ts_tablet_manager.cc:1918] T 3d44db37d9d541c3957bf34c3c79435f P 0269a239f4624610a1ce99d87e3f8967: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 09:01:33.248538 8500 log.cc:1196] T 3d44db37d9d541c3957bf34c3c79435f P 0269a239f4624610a1ce99d87e3f8967: Deleting WAL directory at /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781859658134510-392-0/minicluster-data/ts-2/wal/wals/3d44db37d9d541c3957bf34c3c79435f
I20260619 09:01:33.248729 8500 ts_tablet_manager.cc:1939] T 3d44db37d9d541c3957bf34c3c79435f P 0269a239f4624610a1ce99d87e3f8967: Deleting consensus metadata
I20260619 09:01:33.249475 8499 ts_tablet_manager.cc:1905] T 3d44db37d9d541c3957bf34c3c79435f P f06926dc12064dd78ea8de2e1a4259ed: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 09:01:33.249004 7907 catalog_manager.cc:4841] TS 0269a239f4624610a1ce99d87e3f8967 (127.0.98.3:44393): tablet 3d44db37d9d541c3957bf34c3c79435f (table test-table [id=7f2c7e19489c4ce4b3e4cde1089e72ec]) successfully deleted
I20260619 09:01:33.250881 8499 ts_tablet_manager.cc:1918] T 3d44db37d9d541c3957bf34c3c79435f P f06926dc12064dd78ea8de2e1a4259ed: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 09:01:33.250941 8499 log.cc:1196] T 3d44db37d9d541c3957bf34c3c79435f P f06926dc12064dd78ea8de2e1a4259ed: Deleting WAL directory at /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781859658134510-392-0/minicluster-data/ts-1/wal/wals/3d44db37d9d541c3957bf34c3c79435f
I20260619 09:01:33.251127 8499 ts_tablet_manager.cc:1939] T 3d44db37d9d541c3957bf34c3c79435f P f06926dc12064dd78ea8de2e1a4259ed: Deleting consensus metadata
I20260619 09:01:33.251240 8429 tablet_bootstrap.cc:492] T 89f39eeaacbc4a9497acf8423b97e6b4 P c9d6290881cd499dbaf9dd32d03583aa: No bootstrap required, opened a new log
I20260619 09:01:33.251292 8429 ts_tablet_manager.cc:1397] T 89f39eeaacbc4a9497acf8423b97e6b4 P c9d6290881cd499dbaf9dd32d03583aa: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.001s
I20260619 09:01:33.251436 8429 raft_consensus.cc:348] T 89f39eeaacbc4a9497acf8423b97e6b4 P c9d6290881cd499dbaf9dd32d03583aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9d6290881cd499dbaf9dd32d03583aa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44937 } } peers { permanent_uuid: "f06926dc12064dd78ea8de2e1a4259ed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41951 } } peers { permanent_uuid: "0269a239f4624610a1ce99d87e3f8967" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44393 } }
I20260619 09:01:33.251494 8429 raft_consensus.cc:374] T 89f39eeaacbc4a9497acf8423b97e6b4 P c9d6290881cd499dbaf9dd32d03583aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 09:01:33.251513 8429 raft_consensus.cc:729] T 89f39eeaacbc4a9497acf8423b97e6b4 P c9d6290881cd499dbaf9dd32d03583aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9d6290881cd499dbaf9dd32d03583aa, State: Initialized, Role: FOLLOWER
I20260619 09:01:33.251550 8429 consensus_queue.cc:260] T 89f39eeaacbc4a9497acf8423b97e6b4 P c9d6290881cd499dbaf9dd32d03583aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9d6290881cd499dbaf9dd32d03583aa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44937 } } peers { permanent_uuid: "f06926dc12064dd78ea8de2e1a4259ed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41951 } } peers { permanent_uuid: "0269a239f4624610a1ce99d87e3f8967" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44393 } }
I20260619 09:01:33.251652 8429 ts_tablet_manager.cc:1428] T 89f39eeaacbc4a9497acf8423b97e6b4 P c9d6290881cd499dbaf9dd32d03583aa: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260619 09:01:33.251775 7907 catalog_manager.cc:4841] TS f06926dc12064dd78ea8de2e1a4259ed (127.0.98.2:41951): tablet 3d44db37d9d541c3957bf34c3c79435f (table test-table [id=7f2c7e19489c4ce4b3e4cde1089e72ec]) successfully deleted
I20260619 09:01:33.367333 8436 raft_consensus.cc:482] T 89f39eeaacbc4a9497acf8423b97e6b4 P 0269a239f4624610a1ce99d87e3f8967 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 09:01:33.367448 8436 raft_consensus.cc:504] T 89f39eeaacbc4a9497acf8423b97e6b4 P 0269a239f4624610a1ce99d87e3f8967 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9d6290881cd499dbaf9dd32d03583aa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44937 } } peers { permanent_uuid: "f06926dc12064dd78ea8de2e1a4259ed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41951 } } peers { permanent_uuid: "0269a239f4624610a1ce99d87e3f8967" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44393 } }
I20260619 09:01:33.367575 8436 leader_election.cc:288] T 89f39eeaacbc4a9497acf8423b97e6b4 P 0269a239f4624610a1ce99d87e3f8967 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c9d6290881cd499dbaf9dd32d03583aa (127.0.98.1:44937), f06926dc12064dd78ea8de2e1a4259ed (127.0.98.2:41951)
I20260619 09:01:33.367817 8078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89f39eeaacbc4a9497acf8423b97e6b4" candidate_uuid: "0269a239f4624610a1ce99d87e3f8967" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9d6290881cd499dbaf9dd32d03583aa" is_pre_election: true
I20260619 09:01:33.367851 8218 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89f39eeaacbc4a9497acf8423b97e6b4" candidate_uuid: "0269a239f4624610a1ce99d87e3f8967" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f06926dc12064dd78ea8de2e1a4259ed" is_pre_election: true
I20260619 09:01:33.367918 8078 raft_consensus.cc:2454] T 89f39eeaacbc4a9497acf8423b97e6b4 P c9d6290881cd499dbaf9dd32d03583aa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0269a239f4624610a1ce99d87e3f8967 in term 0.
I20260619 09:01:33.367952 8218 raft_consensus.cc:2454] T 89f39eeaacbc4a9497acf8423b97e6b4 P f06926dc12064dd78ea8de2e1a4259ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0269a239f4624610a1ce99d87e3f8967 in term 0.
I20260619 09:01:33.368084 8291 leader_election.cc:302] T 89f39eeaacbc4a9497acf8423b97e6b4 P 0269a239f4624610a1ce99d87e3f8967 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0269a239f4624610a1ce99d87e3f8967, f06926dc12064dd78ea8de2e1a4259ed; no voters:
I20260619 09:01:33.368182 8436 raft_consensus.cc:2781] T 89f39eeaacbc4a9497acf8423b97e6b4 P 0269a239f4624610a1ce99d87e3f8967 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 09:01:33.368221 8436 raft_consensus.cc:482] T 89f39eeaacbc4a9497acf8423b97e6b4 P 0269a239f4624610a1ce99d87e3f8967 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 09:01:33.368242 8436 raft_consensus.cc:3037] T 89f39eeaacbc4a9497acf8423b97e6b4 P 0269a239f4624610a1ce99d87e3f8967 [term 0 FOLLOWER]: Advancing to term 1
I20260619 09:01:33.368834 8436 raft_consensus.cc:504] T 89f39eeaacbc4a9497acf8423b97e6b4 P 0269a239f4624610a1ce99d87e3f8967 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9d6290881cd499dbaf9dd32d03583aa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44937 } } peers { permanent_uuid: "f06926dc12064dd78ea8de2e1a4259ed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41951 } } peers { permanent_uuid: "0269a239f4624610a1ce99d87e3f8967" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44393 } }
I20260619 09:01:33.368930 8436 leader_election.cc:288] T 89f39eeaacbc4a9497acf8423b97e6b4 P 0269a239f4624610a1ce99d87e3f8967 [CANDIDATE]: Term 1 election: Requested vote from peers c9d6290881cd499dbaf9dd32d03583aa (127.0.98.1:44937), f06926dc12064dd78ea8de2e1a4259ed (127.0.98.2:41951)
I20260619 09:01:33.369093 8078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89f39eeaacbc4a9497acf8423b97e6b4" candidate_uuid: "0269a239f4624610a1ce99d87e3f8967" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9d6290881cd499dbaf9dd32d03583aa"
I20260619 09:01:33.369096 8218 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89f39eeaacbc4a9497acf8423b97e6b4" candidate_uuid: "0269a239f4624610a1ce99d87e3f8967" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f06926dc12064dd78ea8de2e1a4259ed"
I20260619 09:01:33.369154 8078 raft_consensus.cc:3037] T 89f39eeaacbc4a9497acf8423b97e6b4 P c9d6290881cd499dbaf9dd32d03583aa [term 0 FOLLOWER]: Advancing to term 1
I20260619 09:01:33.369161 8218 raft_consensus.cc:3037] T 89f39eeaacbc4a9497acf8423b97e6b4 P f06926dc12064dd78ea8de2e1a4259ed [term 0 FOLLOWER]: Advancing to term 1
I20260619 09:01:33.369704 8078 raft_consensus.cc:2454] T 89f39eeaacbc4a9497acf8423b97e6b4 P c9d6290881cd499dbaf9dd32d03583aa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0269a239f4624610a1ce99d87e3f8967 in term 1.
I20260619 09:01:33.369765 8218 raft_consensus.cc:2454] T 89f39eeaacbc4a9497acf8423b97e6b4 P f06926dc12064dd78ea8de2e1a4259ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0269a239f4624610a1ce99d87e3f8967 in term 1.
I20260619 09:01:33.369875 8291 leader_election.cc:302] T 89f39eeaacbc4a9497acf8423b97e6b4 P 0269a239f4624610a1ce99d87e3f8967 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0269a239f4624610a1ce99d87e3f8967, c9d6290881cd499dbaf9dd32d03583aa; no voters:
I20260619 09:01:33.369956 8436 raft_consensus.cc:2781] T 89f39eeaacbc4a9497acf8423b97e6b4 P 0269a239f4624610a1ce99d87e3f8967 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 09:01:33.370007 8436 raft_consensus.cc:686] T 89f39eeaacbc4a9497acf8423b97e6b4 P 0269a239f4624610a1ce99d87e3f8967 [term 1 LEADER]: Becoming Leader. State: Replica: 0269a239f4624610a1ce99d87e3f8967, State: Running, Role: LEADER
I20260619 09:01:33.370054 8436 consensus_queue.cc:237] T 89f39eeaacbc4a9497acf8423b97e6b4 P 0269a239f4624610a1ce99d87e3f8967 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9d6290881cd499dbaf9dd32d03583aa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44937 } } peers { permanent_uuid: "f06926dc12064dd78ea8de2e1a4259ed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41951 } } peers { permanent_uuid: "0269a239f4624610a1ce99d87e3f8967" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44393 } }
I20260619 09:01:33.370622 7920 catalog_manager.cc:5495] T 89f39eeaacbc4a9497acf8423b97e6b4 P 0269a239f4624610a1ce99d87e3f8967 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0269a239f4624610a1ce99d87e3f8967 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "0269a239f4624610a1ce99d87e3f8967" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9d6290881cd499dbaf9dd32d03583aa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44937 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f06926dc12064dd78ea8de2e1a4259ed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41951 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0269a239f4624610a1ce99d87e3f8967" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44393 } health_report { overall_health: HEALTHY } } }
I20260619 09:01:33.393584 8485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:33.391260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38162 (local address 127.0.98.3:44393)
0619 09:01:33.391307 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:33.391308 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:33.391317 (+ 9us) server_negotiation.cc:407] Connection header received
0619 09:01:33.391339 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:33.391340 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:33.391349 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:33.391410 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 09:01:33.391491 (+ 81us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:33.392220 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:33.392931 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:33.393237 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:33.393299 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:33.393396 (+ 97us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 09:01:33.393427 (+ 31us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 09:01:33.393470 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0619 09:01:33.393505 (+ 35us) server_negotiation.cc:299] Negotiation successful
0619 09:01:33.393522 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260619 09:01:33.395403 8218 raft_consensus.cc:1261] T 89f39eeaacbc4a9497acf8423b97e6b4 P f06926dc12064dd78ea8de2e1a4259ed [term 1 FOLLOWER]: Refusing update from remote peer 0269a239f4624610a1ce99d87e3f8967: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 09:01:33.395443 8078 raft_consensus.cc:1261] T 89f39eeaacbc4a9497acf8423b97e6b4 P c9d6290881cd499dbaf9dd32d03583aa [term 1 FOLLOWER]: Refusing update from remote peer 0269a239f4624610a1ce99d87e3f8967: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 09:01:33.395588 8458 consensus_queue.cc:1035] T 89f39eeaacbc4a9497acf8423b97e6b4 P 0269a239f4624610a1ce99d87e3f8967 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f06926dc12064dd78ea8de2e1a4259ed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 09:01:33.395695 8436 consensus_queue.cc:1035] T 89f39eeaacbc4a9497acf8423b97e6b4 P 0269a239f4624610a1ce99d87e3f8967 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9d6290881cd499dbaf9dd32d03583aa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 09:01:33.404826 7919 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33968:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 09:01:33.404908 7919 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33968:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 09:01:33.405387 7919 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 75e0d761165f4e3f9aa72ac34214c70c: Sending DeleteTablet for 3 replicas of tablet 89f39eeaacbc4a9497acf8423b97e6b4
I20260619 09:01:33.405593 8058 tablet_service.cc:1514] Processing DeleteTablet for tablet 89f39eeaacbc4a9497acf8423b97e6b4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 09:01:33 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48302
I20260619 09:01:33.405613 8198 tablet_service.cc:1514] Processing DeleteTablet for tablet 89f39eeaacbc4a9497acf8423b97e6b4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 09:01:33 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51328
I20260619 09:01:33.405689 8498 tablet_replica.cc:330] T 89f39eeaacbc4a9497acf8423b97e6b4 P c9d6290881cd499dbaf9dd32d03583aa: stopping tablet replica
I20260619 09:01:33.405750 8498 raft_consensus.cc:2229] T 89f39eeaacbc4a9497acf8423b97e6b4 P c9d6290881cd499dbaf9dd32d03583aa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 09:01:33.405750 8338 tablet_service.cc:1514] Processing DeleteTablet for tablet 89f39eeaacbc4a9497acf8423b97e6b4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 09:01:33 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38158
I20260619 09:01:33.405782 8498 raft_consensus.cc:2258] T 89f39eeaacbc4a9497acf8423b97e6b4 P c9d6290881cd499dbaf9dd32d03583aa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 09:01:33.405825 8500 tablet_replica.cc:330] T 89f39eeaacbc4a9497acf8423b97e6b4 P 0269a239f4624610a1ce99d87e3f8967: stopping tablet replica
I20260619 09:01:33.405879 8500 raft_consensus.cc:2229] T 89f39eeaacbc4a9497acf8423b97e6b4 P 0269a239f4624610a1ce99d87e3f8967 [term 1 LEADER]: Raft consensus shutting down.
I20260619 09:01:33.405932 8500 raft_consensus.cc:2258] T 89f39eeaacbc4a9497acf8423b97e6b4 P 0269a239f4624610a1ce99d87e3f8967 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 09:01:33.405936 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ez5NB/build/release/bin/kudu with pid 7993
I20260619 09:01:33.406159 8498 ts_tablet_manager.cc:1905] T 89f39eeaacbc4a9497acf8423b97e6b4 P c9d6290881cd499dbaf9dd32d03583aa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 09:01:33.407469 8499 tablet_replica.cc:330] T 89f39eeaacbc4a9497acf8423b97e6b4 P f06926dc12064dd78ea8de2e1a4259ed: stopping tablet replica
I20260619 09:01:33.407547 8499 raft_consensus.cc:2229] T 89f39eeaacbc4a9497acf8423b97e6b4 P f06926dc12064dd78ea8de2e1a4259ed [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 09:01:33.407588 8499 raft_consensus.cc:2258] T 89f39eeaacbc4a9497acf8423b97e6b4 P f06926dc12064dd78ea8de2e1a4259ed [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 09:01:33.407799 8500 ts_tablet_manager.cc:1905] T 89f39eeaacbc4a9497acf8423b97e6b4 P 0269a239f4624610a1ce99d87e3f8967: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 09:01:33.407871 8499 ts_tablet_manager.cc:1905] T 89f39eeaacbc4a9497acf8423b97e6b4 P f06926dc12064dd78ea8de2e1a4259ed: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 09:01:33.408999 8500 ts_tablet_manager.cc:1918] T 89f39eeaacbc4a9497acf8423b97e6b4 P 0269a239f4624610a1ce99d87e3f8967: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 09:01:33.409046 8500 log.cc:1196] T 89f39eeaacbc4a9497acf8423b97e6b4 P 0269a239f4624610a1ce99d87e3f8967: Deleting WAL directory at /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781859658134510-392-0/minicluster-data/ts-2/wal/wals/89f39eeaacbc4a9497acf8423b97e6b4
I20260619 09:01:33.409209 8500 ts_tablet_manager.cc:1939] T 89f39eeaacbc4a9497acf8423b97e6b4 P 0269a239f4624610a1ce99d87e3f8967: Deleting consensus metadata
I20260619 09:01:33.409371 8499 ts_tablet_manager.cc:1918] T 89f39eeaacbc4a9497acf8423b97e6b4 P f06926dc12064dd78ea8de2e1a4259ed: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 09:01:33.409425 8499 log.cc:1196] T 89f39eeaacbc4a9497acf8423b97e6b4 P f06926dc12064dd78ea8de2e1a4259ed: Deleting WAL directory at /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781859658134510-392-0/minicluster-data/ts-1/wal/wals/89f39eeaacbc4a9497acf8423b97e6b4
I20260619 09:01:33.409462 7907 catalog_manager.cc:4841] TS 0269a239f4624610a1ce99d87e3f8967 (127.0.98.3:44393): tablet 89f39eeaacbc4a9497acf8423b97e6b4 (table test-table [id=873634747b5448429c2aa16f5f44deeb]) successfully deleted
I20260619 09:01:33.409600 8499 ts_tablet_manager.cc:1939] T 89f39eeaacbc4a9497acf8423b97e6b4 P f06926dc12064dd78ea8de2e1a4259ed: Deleting consensus metadata
I20260619 09:01:33.410022 7907 catalog_manager.cc:4841] TS f06926dc12064dd78ea8de2e1a4259ed (127.0.98.2:41951): tablet 89f39eeaacbc4a9497acf8423b97e6b4 (table test-table [id=873634747b5448429c2aa16f5f44deeb]) successfully deleted
W20260619 09:01:33.411959 7907 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:44937 (error 108)
I20260619 09:01:33.412258 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ez5NB/build/release/bin/kudu with pid 8133
I20260619 09:01:33.412957 8527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:33.412327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44937 (local address 127.0.0.1:48340)
0619 09:01:33.412873 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0619 09:01:33.412921 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":523,"thread_start_us":65,"threads_started":1}
W20260619 09:01:33.413064 7907 catalog_manager.cc:4584] TS c9d6290881cd499dbaf9dd32d03583aa (127.0.98.1:44937): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 89f39eeaacbc4a9497acf8423b97e6b4: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44937: connect: Connection refused (error 111)
I20260619 09:01:33.417065 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ez5NB/build/release/bin/kudu with pid 8273
I20260619 09:01:33.421607 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ez5NB/build/release/bin/kudu with pid 7889
2026-06-19T09:01:33Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (2669 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781859658134510-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 09:01:33 dist-test-slave-mtdr krb5kdc[8530](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 09:01:33 dist-test-slave-mtdr krb5kdc[8530](info): set up 2 sockets
Jun 19 09:01:33 dist-test-slave-mtdr krb5kdc[8530](info): commencing operation
krb5kdc: starting...
W20260619 09:01:34.367103 392 mini_kdc.cc:121] Time spent starting KDC: real 0.926s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781859658134510-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781859658134510-392-0/krb5kdc/test-user.keytab.
Jun 19 09:01:34 dist-test-slave-mtdr krb5kdc[8530](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859694, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T09:01:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T09:01:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781859658134510-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781859658134510-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 09:01:34.476775 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ez5NB/build/release/bin/kudu
/tmp/dist-test-task4ez5NB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781859658134510-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781859658134510-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781859658134510-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781859658134510-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42831
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781859658134510-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40179
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42831
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781859658134510-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781859658134510-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781859658134510-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 09:01:34.552719 8543 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 09:01:34.552922 8543 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 09:01:34.552953 8543 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 09:01:34.554090 8543 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 09:01:34.554129 8543 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 09:01:34.554147 8543 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 09:01:34.554164 8543 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 09:01:34.554179 8543 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 09:01:34.555675 8543 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40179
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781859658134510-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781859658134510-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42831
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781859658134510-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42831
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781859658134510-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8543
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781859658134510-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 09:01:34.555903 8543 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 09:01:34.556097 8543 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 09:01:34.560694 8551 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:01:34.560823 8543 server_base.cc:1034] running on GCE node
W20260619 09:01:34.560736 8548 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:01:34.560704 8549 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:01:34.561242 8543 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 09:01:34.561486 8543 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 09:01:34.562633 8543 hybrid_clock.cc:648] HybridClock initialized: now 1781859694562619 us; error 26 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-06-19T09:01:34Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1142 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859658134510-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 09:01:34 dist-test-slave-mtdr krb5kdc[8556](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 09:01:34 dist-test-slave-mtdr krb5kdc[8556](info): set up 2 sockets
Jun 19 09:01:34 dist-test-slave-mtdr krb5kdc[8556](info): commencing operation
krb5kdc: starting...
W20260619 09:01:35.515085 392 mini_kdc.cc:121] Time spent starting KDC: real 0.932s user 0.003s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859658134510-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859658134510-392-0/krb5kdc/test-user.keytab.
Jun 19 09:01:35 dist-test-slave-mtdr krb5kdc[8556](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859695, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T09:01:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T09:01:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859658134510-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859658134510-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 09:01:35.625293 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ez5NB/build/release/bin/kudu
/tmp/dist-test-task4ez5NB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859658134510-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859658134510-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859658134510-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859658134510-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34829
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859658134510-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34829
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859658134510-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859658134510-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859658134510-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 09:01:35.704970 8569 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 09:01:35.705181 8569 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 09:01:35.705211 8569 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 09:01:35.706342 8569 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 09:01:35.706382 8569 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 09:01:35.706399 8569 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 09:01:35.706415 8569 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 09:01:35.706430 8569 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 09:01:35.707928 8569 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859658134510-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859658134510-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34829
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859658134510-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34829
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859658134510-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8569
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859658134510-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 09:01:35.708163 8569 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 09:01:35.708355 8569 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 09:01:35.712711 8574 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:01:35.712796 8577 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:01:35.712834 8575 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:01:35.713030 8569 server_base.cc:1034] running on GCE node
I20260619 09:01:35.713251 8569 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 09:01:35.713517 8569 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 09:01:35.714664 8569 hybrid_clock.cc:648] HybridClock initialized: now 1781859695714647 us; error 34 us; skew 500 ppm
Jun 19 09:01:35 dist-test-slave-mtdr krb5kdc[8556](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859695, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 09:01:35.720512 8569 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 09:01:35.720935 8569 webserver.cc:460] Webserver started at http://127.0.98.62:45893/ using document root <none> and password file <none>
I20260619 09:01:35.721134 8569 fs_manager.cc:359] Metadata directory not provided
I20260619 09:01:35.721176 8569 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 09:01:35.721278 8569 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 09:01:35.722276 8569 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859658134510-392-0/minicluster-data/master-0/data/instance:
uuid: "22161c26406a4056857a369b1a64c600"
format_stamp: "Formatted at 2026-06-19 09:01:35 on dist-test-slave-mtdr"
server_key: "14570fa236eeec33262a5be827374ba1"
server_key_iv: "87460b10b23dcb9a3fbb437f52b8d98f"
server_key_version: "encryptionkey@0"
I20260619 09:01:35.722597 8569 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859658134510-392-0/minicluster-data/master-0/wal/instance:
uuid: "22161c26406a4056857a369b1a64c600"
format_stamp: "Formatted at 2026-06-19 09:01:35 on dist-test-slave-mtdr"
server_key: "14570fa236eeec33262a5be827374ba1"
server_key_iv: "87460b10b23dcb9a3fbb437f52b8d98f"
server_key_version: "encryptionkey@0"
I20260619 09:01:35.723892 8569 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260619 09:01:35.724668 8584 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 09:01:35.724874 8569 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 09:01:35.724973 8569 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859658134510-392-0/minicluster-data/master-0/data,/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859658134510-392-0/minicluster-data/master-0/wal
uuid: "22161c26406a4056857a369b1a64c600"
format_stamp: "Formatted at 2026-06-19 09:01:35 on dist-test-slave-mtdr"
server_key: "14570fa236eeec33262a5be827374ba1"
server_key_iv: "87460b10b23dcb9a3fbb437f52b8d98f"
server_key_version: "encryptionkey@0"
I20260619 09:01:35.725044 8569 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859658134510-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859658134510-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859658134510-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 09:01:35.741888 8569 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 09:01:35.743897 8569 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 09:01:35.744036 8569 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 09:01:35.748229 8569 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34829
I20260619 09:01:35.748261 8636 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34829 every 8 connection(s)
I20260619 09:01:35.748749 8569 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859658134510-392-0/minicluster-data/master-0/data/info.pb
I20260619 09:01:35.749372 8637 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 09:01:35.749670 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ez5NB/build/release/bin/kudu as pid 8569
I20260619 09:01:35.749780 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859658134510-392-0/minicluster-data/master-0/wal/instance
I20260619 09:01:35.749964 392 external_mini_cluster.cc:1428] Setting key 3e7d25881cc4c6190c0071c20d1d618b
I20260619 09:01:35.752236 8637 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 22161c26406a4056857a369b1a64c600: Bootstrap starting.
I20260619 09:01:35.752797 8637 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 22161c26406a4056857a369b1a64c600: Neither blocks nor log segments found. Creating new log.
I20260619 09:01:35.753048 8637 log.cc:824] T 00000000000000000000000000000000 P 22161c26406a4056857a369b1a64c600: Log is configured to *not* fsync() on all Append() calls
I20260619 09:01:35.753724 8637 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 22161c26406a4056857a369b1a64c600: No bootstrap required, opened a new log
I20260619 09:01:35.755964 8637 raft_consensus.cc:348] T 00000000000000000000000000000000 P 22161c26406a4056857a369b1a64c600 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22161c26406a4056857a369b1a64c600" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34829 } }
I20260619 09:01:35.756098 8637 raft_consensus.cc:374] T 00000000000000000000000000000000 P 22161c26406a4056857a369b1a64c600 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 09:01:35.756119 8637 raft_consensus.cc:729] T 00000000000000000000000000000000 P 22161c26406a4056857a369b1a64c600 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22161c26406a4056857a369b1a64c600, State: Initialized, Role: FOLLOWER
I20260619 09:01:35.756224 8637 consensus_queue.cc:260] T 00000000000000000000000000000000 P 22161c26406a4056857a369b1a64c600 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22161c26406a4056857a369b1a64c600" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34829 } }
I20260619 09:01:35.756291 8637 raft_consensus.cc:388] T 00000000000000000000000000000000 P 22161c26406a4056857a369b1a64c600 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 09:01:35.756312 8637 raft_consensus.cc:482] T 00000000000000000000000000000000 P 22161c26406a4056857a369b1a64c600 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 09:01:35.756356 8637 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 22161c26406a4056857a369b1a64c600 [term 0 FOLLOWER]: Advancing to term 1
Jun 19 09:01:35 dist-test-slave-mtdr krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859695, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 09:01:35.756877 8637 raft_consensus.cc:504] T 00000000000000000000000000000000 P 22161c26406a4056857a369b1a64c600 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22161c26406a4056857a369b1a64c600" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34829 } }
I20260619 09:01:35.756978 8637 leader_election.cc:302] T 00000000000000000000000000000000 P 22161c26406a4056857a369b1a64c600 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 22161c26406a4056857a369b1a64c600; no voters:
I20260619 09:01:35.757151 8637 leader_election.cc:288] T 00000000000000000000000000000000 P 22161c26406a4056857a369b1a64c600 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 09:01:35.757191 8642 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 22161c26406a4056857a369b1a64c600 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 09:01:35.757344 8642 raft_consensus.cc:686] T 00000000000000000000000000000000 P 22161c26406a4056857a369b1a64c600 [term 1 LEADER]: Becoming Leader. State: Replica: 22161c26406a4056857a369b1a64c600, State: Running, Role: LEADER
I20260619 09:01:35.757397 8637 sys_catalog.cc:564] T 00000000000000000000000000000000 P 22161c26406a4056857a369b1a64c600 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 09:01:35.757498 8642 consensus_queue.cc:237] T 00000000000000000000000000000000 P 22161c26406a4056857a369b1a64c600 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22161c26406a4056857a369b1a64c600" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34829 } }
I20260619 09:01:35.757915 8643 sys_catalog.cc:455] T 00000000000000000000000000000000 P 22161c26406a4056857a369b1a64c600 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "22161c26406a4056857a369b1a64c600" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22161c26406a4056857a369b1a64c600" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34829 } } }
I20260619 09:01:35.758150 8643 sys_catalog.cc:458] T 00000000000000000000000000000000 P 22161c26406a4056857a369b1a64c600 [sys.catalog]: This master's current role is: LEADER
I20260619 09:01:35.758081 8644 sys_catalog.cc:455] T 00000000000000000000000000000000 P 22161c26406a4056857a369b1a64c600 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 22161c26406a4056857a369b1a64c600. Latest consensus state: current_term: 1 leader_uuid: "22161c26406a4056857a369b1a64c600" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22161c26406a4056857a369b1a64c600" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34829 } } }
I20260619 09:01:35.758388 8644 sys_catalog.cc:458] T 00000000000000000000000000000000 P 22161c26406a4056857a369b1a64c600 [sys.catalog]: This master's current role is: LEADER
I20260619 09:01:35.758486 8652 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 09:01:35.759013 8652 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 09:01:35.760495 8640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:35.750490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40990 (local address 127.0.98.62:34829)
0619 09:01:35.750655 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:35.750664 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:35.750688 (+ 24us) server_negotiation.cc:407] Connection header received
0619 09:01:35.751123 (+ 435us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:35.751129 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:35.751177 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:35.751289 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:01:35.752054 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:35.752975 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:35.753894 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:35.754287 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:35.754365 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:35.756887 (+ 2522us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:01:35.756892 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:01:35.756892 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:01:35.756904 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:01:35.758860 (+ 1956us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:01:35.759241 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:01:35.759244 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:01:35.759244 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:01:35.759330 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:01:35.759505 (+ 175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:01:35.759506 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:01:35.759506 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:01:35.759786 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:01:35.759929 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0619 09:01:35.760198 (+ 269us) server_negotiation.cc:299] Negotiation successful
0619 09:01:35.760331 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":62,"threads_started":1}
I20260619 09:01:35.760526 8652 catalog_manager.cc:1347] Generated new cluster ID: 2a52b1c37c9c471da64ab3c9b6e20e89
I20260619 09:01:35.760577 8652 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 09:01:35.779421 8652 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 09:01:35.779934 8652 catalog_manager.cc:1504] Loading token signing keys...
I20260619 09:01:35.787264 8652 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 22161c26406a4056857a369b1a64c600: Generated new TSK 0
I20260619 09:01:35.787429 8652 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859658134510-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859658134510-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 09:01:35.814209 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ez5NB/build/release/bin/kudu
/tmp/dist-test-task4ez5NB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859658134510-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859658134510-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859658134510-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859658134510-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34829
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859658134510-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859658134510-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859658134510-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859658134510-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 09:01:35.893613 8663 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 09:01:35.893813 8663 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 09:01:35.893846 8663 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 09:01:35.894996 8663 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 09:01:35.895035 8663 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 09:01:35.895071 8663 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 09:01:35.896540 8663 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859658134510-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859658134510-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859658134510-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859658134510-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34829
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8663
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859658134510-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 09:01:35.896763 8663 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 09:01:35.896960 8663 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 09:01:35.901558 8671 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:01:35.901655 8663 server_base.cc:1034] running on GCE node
W20260619 09:01:35.901687 8668 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:01:35.901526 8669 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:01:35.901993 8663 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 09:01:35.902231 8663 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 09:01:35.903366 8663 hybrid_clock.cc:648] HybridClock initialized: now 1781859695903353 us; error 26 us; skew 500 ppm
Jun 19 09:01:35 dist-test-slave-mtdr krb5kdc[8556](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859695, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 09:01:35.905814 8663 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 09:01:35.906208 8663 webserver.cc:460] Webserver started at http://127.0.98.1:37433/ using document root <none> and password file <none>
I20260619 09:01:35.906405 8663 fs_manager.cc:359] Metadata directory not provided
I20260619 09:01:35.906451 8663 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 09:01:35.906599 8663 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 09:01:35.907619 8663 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859658134510-392-0/minicluster-data/ts-0/data/instance:
uuid: "557ed5c591424db0b4456ca89ef08d6a"
format_stamp: "Formatted at 2026-06-19 09:01:35 on dist-test-slave-mtdr"
server_key: "f7a4f9d4facf04911a81f14b547e39cb"
server_key_iv: "86b0d69ab74c67b323cb129532e0ff82"
server_key_version: "encryptionkey@0"
I20260619 09:01:35.907930 8663 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859658134510-392-0/minicluster-data/ts-0/wal/instance:
uuid: "557ed5c591424db0b4456ca89ef08d6a"
format_stamp: "Formatted at 2026-06-19 09:01:35 on dist-test-slave-mtdr"
server_key: "f7a4f9d4facf04911a81f14b547e39cb"
server_key_iv: "86b0d69ab74c67b323cb129532e0ff82"
server_key_version: "encryptionkey@0"
I20260619 09:01:35.909071 8663 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260619 09:01:35.909708 8678 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 09:01:35.909837 8663 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 09:01:35.909917 8663 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859658134510-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859658134510-392-0/minicluster-data/ts-0/wal
uuid: "557ed5c591424db0b4456ca89ef08d6a"
format_stamp: "Formatted at 2026-06-19 09:01:35 on dist-test-slave-mtdr"
server_key: "f7a4f9d4facf04911a81f14b547e39cb"
server_key_iv: "86b0d69ab74c67b323cb129532e0ff82"
server_key_version: "encryptionkey@0"
I20260619 09:01:35.909981 8663 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859658134510-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859658134510-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859658134510-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 09:01:35.921362 8663 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 09:01:35.923310 8663 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 09:01:35.923435 8663 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 09:01:35.923633 8663 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 09:01:35.923910 8663 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 09:01:35.923944 8663 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 09:01:35.923971 8663 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 09:01:35.923996 8663 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 09:01:35.929360 8663 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35029
I20260619 09:01:35.929414 8791 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35029 every 8 connection(s)
I20260619 09:01:35.929703 8663 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859658134510-392-0/minicluster-data/ts-0/data/info.pb
Jun 19 09:01:35 dist-test-slave-mtdr krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859695, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 09:01:35.938376 8640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:35.930308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49759 (local address 127.0.98.62:34829)
0619 09:01:35.930379 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:35.930381 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:35.930621 (+ 240us) server_negotiation.cc:407] Connection header received
0619 09:01:35.931591 (+ 970us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:35.931592 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:35.931604 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:35.931671 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:01:35.932215 (+ 544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:35.932737 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:35.933758 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:35.933967 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:35.934047 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:35.936090 (+ 2043us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:01:35.936091 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:01:35.936091 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:01:35.936098 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:01:35.937293 (+ 1195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:01:35.937618 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:01:35.937618 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:01:35.937619 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:01:35.937664 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:01:35.937840 (+ 176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:01:35.937840 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:01:35.937841 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:01:35.937931 (+ 90us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:01:35.937975 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0619 09:01:35.938250 (+ 275us) server_negotiation.cc:299] Negotiation successful
0619 09:01:35.938312 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260619 09:01:35.938555 8794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:35.930294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34829 (local address 127.0.98.1:49759)
0619 09:01:35.930441 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0619 09:01:35.930474 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0619 09:01:35.931465 (+ 991us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:01:35.931717 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:01:35.931720 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:01:35.931952 (+ 232us) client_negotiation.cc:189] Negotiated authn=SASL
0619 09:01:35.932138 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:35.932142 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:35.932833 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:35.932834 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:35.933662 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:35.933664 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:35.934049 (+ 385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:35.934050 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:35.934157 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:35.934587 (+ 430us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 09:01:35.934596 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0619 09:01:35.936009 (+ 1413us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 09:01:35.937359 (+ 1350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:01:35.937364 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:01:35.937366 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:01:35.937562 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:01:35.937708 (+ 146us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:01:35.937708 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:01:35.937709 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:01:35.937798 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:01:35.938005 (+ 207us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 09:01:35.938011 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 09:01:35.938143 (+ 132us) client_negotiation.cc:769] Sending connection context
0619 09:01:35.938233 (+ 90us) client_negotiation.cc:240] Negotiation successful
0619 09:01:35.938350 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":84,"threads_started":1}
I20260619 09:01:35.939023 8792 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34829
I20260619 09:01:35.939109 8792 heartbeater.cc:464] Registering TS with master...
I20260619 09:01:35.939271 8792 heartbeater.cc:511] Master 127.0.98.62:34829 requested a full tablet report, sending...
I20260619 09:01:35.939342 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ez5NB/build/release/bin/kudu as pid 8663
I20260619 09:01:35.939441 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859658134510-392-0/minicluster-data/ts-0/wal/instance
I20260619 09:01:35.939639 392 external_mini_cluster.cc:1428] Setting key dd8ed3fed0e52ebb30abdb617e5413e1
I20260619 09:01:35.939805 8601 ts_manager.cc:194] Registered new tserver with Master: 557ed5c591424db0b4456ca89ef08d6a (127.0.98.1:35029)
I20260619 09:01:35.940717 8601 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:49759
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859658134510-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859658134510-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 09:01:35.961396 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ez5NB/build/release/bin/kudu
/tmp/dist-test-task4ez5NB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859658134510-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859658134510-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859658134510-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859658134510-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34829
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859658134510-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859658134510-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859658134510-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859658134510-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 09:01:36.040215 8797 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 09:01:36.040423 8797 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 09:01:36.040445 8797 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 09:01:36.041585 8797 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 09:01:36.041625 8797 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 09:01:36.041649 8797 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 09:01:36.043098 8797 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859658134510-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859658134510-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859658134510-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859658134510-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34829
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8797
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859658134510-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 09:01:36.043291 8797 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 09:01:36.043463 8797 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 09:01:36.047987 8802 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:01:36.047999 8803 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:01:36.048009 8805 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:01:36.048256 8797 server_base.cc:1034] running on GCE node
I20260619 09:01:36.048457 8797 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 09:01:36.048666 8797 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 09:01:36.049821 8797 hybrid_clock.cc:648] HybridClock initialized: now 1781859696049792 us; error 31 us; skew 500 ppm
Jun 19 09:01:36 dist-test-slave-mtdr krb5kdc[8556](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859696, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 09:01:36.052145 8797 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 09:01:36.052500 8797 webserver.cc:460] Webserver started at http://127.0.98.2:37635/ using document root <none> and password file <none>
I20260619 09:01:36.052662 8797 fs_manager.cc:359] Metadata directory not provided
I20260619 09:01:36.052697 8797 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 09:01:36.052793 8797 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 09:01:36.053716 8797 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859658134510-392-0/minicluster-data/ts-1/data/instance:
uuid: "b1e5855d7c84416d977defe417c4ebf5"
format_stamp: "Formatted at 2026-06-19 09:01:36 on dist-test-slave-mtdr"
server_key: "dfa264e6c1611491b9d8e891f9eb911d"
server_key_iv: "cd722e0299be4dc706e3890f6844f4a4"
server_key_version: "encryptionkey@0"
I20260619 09:01:36.054001 8797 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859658134510-392-0/minicluster-data/ts-1/wal/instance:
uuid: "b1e5855d7c84416d977defe417c4ebf5"
format_stamp: "Formatted at 2026-06-19 09:01:36 on dist-test-slave-mtdr"
server_key: "dfa264e6c1611491b9d8e891f9eb911d"
server_key_iv: "cd722e0299be4dc706e3890f6844f4a4"
server_key_version: "encryptionkey@0"
I20260619 09:01:36.055136 8797 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260619 09:01:36.055771 8812 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 09:01:36.055910 8797 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 09:01:36.055981 8797 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859658134510-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859658134510-392-0/minicluster-data/ts-1/wal
uuid: "b1e5855d7c84416d977defe417c4ebf5"
format_stamp: "Formatted at 2026-06-19 09:01:36 on dist-test-slave-mtdr"
server_key: "dfa264e6c1611491b9d8e891f9eb911d"
server_key_iv: "cd722e0299be4dc706e3890f6844f4a4"
server_key_version: "encryptionkey@0"
I20260619 09:01:36.056041 8797 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859658134510-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859658134510-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859658134510-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 09:01:36.065960 8797 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 09:01:36.067814 8797 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 09:01:36.067926 8797 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 09:01:36.068125 8797 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 09:01:36.068384 8797 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 09:01:36.068415 8797 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 09:01:36.068437 8797 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 09:01:36.068460 8797 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 09:01:36.073606 8797 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37743
I20260619 09:01:36.073658 8925 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37743 every 8 connection(s)
I20260619 09:01:36.073935 8797 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859658134510-392-0/minicluster-data/ts-1/data/info.pb
I20260619 09:01:36.075711 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ez5NB/build/release/bin/kudu as pid 8797
I20260619 09:01:36.075834 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859658134510-392-0/minicluster-data/ts-1/wal/instance
I20260619 09:01:36.076040 392 external_mini_cluster.cc:1428] Setting key f5884ecceb4b3ebb93f2c2bbd3c1bb37
Jun 19 09:01:36 dist-test-slave-mtdr krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859696, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 09:01:36.083721 8640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:36.074627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36627 (local address 127.0.98.62:34829)
0619 09:01:36.074705 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:36.074707 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:36.074885 (+ 178us) server_negotiation.cc:407] Connection header received
0619 09:01:36.076017 (+ 1132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:36.076018 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:36.076028 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:36.076065 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:01:36.076807 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:36.077277 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:36.078504 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:36.078786 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:36.078866 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:36.080969 (+ 2103us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:01:36.080971 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:01:36.080971 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:01:36.080979 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:01:36.082391 (+ 1412us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:01:36.082839 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:01:36.082840 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:01:36.082841 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:01:36.082899 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:01:36.083118 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:01:36.083119 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:01:36.083120 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:01:36.083252 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:01:36.083318 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0619 09:01:36.083541 (+ 223us) server_negotiation.cc:299] Negotiation successful
0619 09:01:36.083628 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260619 09:01:36.083781 8928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:36.074579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34829 (local address 127.0.98.2:36627)
0619 09:01:36.074751 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0619 09:01:36.074789 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 09:01:36.075913 (+ 1124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:01:36.076162 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:01:36.076167 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:01:36.076472 (+ 305us) client_negotiation.cc:189] Negotiated authn=SASL
0619 09:01:36.076716 (+ 244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:36.076723 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:36.077369 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:36.077370 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:36.078424 (+ 1054us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:36.078426 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:36.078902 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:36.078903 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:36.079046 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:36.079524 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 09:01:36.079533 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0619 09:01:36.080886 (+ 1353us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 09:01:36.082481 (+ 1595us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:01:36.082487 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:01:36.082489 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:01:36.082720 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:01:36.082962 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:01:36.082963 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:01:36.082963 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:01:36.083059 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:01:36.083313 (+ 254us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 09:01:36.083316 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 09:01:36.083424 (+ 108us) client_negotiation.cc:769] Sending connection context
0619 09:01:36.083499 (+ 75us) client_negotiation.cc:240] Negotiation successful
0619 09:01:36.083613 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":77,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260619 09:01:36.084390 8926 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34829
I20260619 09:01:36.084468 8926 heartbeater.cc:464] Registering TS with master...
I20260619 09:01:36.084605 8926 heartbeater.cc:511] Master 127.0.98.62:34829 requested a full tablet report, sending...
I20260619 09:01:36.084950 8601 ts_manager.cc:194] Registered new tserver with Master: b1e5855d7c84416d977defe417c4ebf5 (127.0.98.2:37743)
I20260619 09:01:36.085297 8601 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:36627
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859658134510-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859658134510-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 09:01:36.099419 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ez5NB/build/release/bin/kudu
/tmp/dist-test-task4ez5NB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859658134510-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859658134510-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859658134510-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859658134510-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34829
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859658134510-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859658134510-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859658134510-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859658134510-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 09:01:36.177414 8931 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 09:01:36.177611 8931 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 09:01:36.177634 8931 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 09:01:36.178740 8931 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 09:01:36.178773 8931 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 09:01:36.178797 8931 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 09:01:36.180382 8931 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859658134510-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859658134510-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859658134510-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859658134510-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34829
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8931
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859658134510-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 09:01:36.180568 8931 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 09:01:36.180742 8931 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 09:01:36.185446 8937 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:01:36.185436 8939 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:01:36.185619 8931 server_base.cc:1034] running on GCE node
W20260619 09:01:36.185438 8936 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:01:36.185912 8931 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 09:01:36.186137 8931 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 09:01:36.187283 8931 hybrid_clock.cc:648] HybridClock initialized: now 1781859696187275 us; error 31 us; skew 500 ppm
Jun 19 09:01:36 dist-test-slave-mtdr krb5kdc[8556](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859696, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 09:01:36.189657 8931 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 09:01:36.190130 8931 webserver.cc:460] Webserver started at http://127.0.98.3:38665/ using document root <none> and password file <none>
I20260619 09:01:36.190380 8931 fs_manager.cc:359] Metadata directory not provided
I20260619 09:01:36.190434 8931 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 09:01:36.190557 8931 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 09:01:36.191594 8931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859658134510-392-0/minicluster-data/ts-2/data/instance:
uuid: "5a350609debf460e84a8cdcbb37f0e5c"
format_stamp: "Formatted at 2026-06-19 09:01:36 on dist-test-slave-mtdr"
server_key: "226e48a95a242b4cbf174ddd2511e57e"
server_key_iv: "78d8a042da4fd84eed1cb29c653aef55"
server_key_version: "encryptionkey@0"
I20260619 09:01:36.191913 8931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859658134510-392-0/minicluster-data/ts-2/wal/instance:
uuid: "5a350609debf460e84a8cdcbb37f0e5c"
format_stamp: "Formatted at 2026-06-19 09:01:36 on dist-test-slave-mtdr"
server_key: "226e48a95a242b4cbf174ddd2511e57e"
server_key_iv: "78d8a042da4fd84eed1cb29c653aef55"
server_key_version: "encryptionkey@0"
I20260619 09:01:36.193099 8931 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260619 09:01:36.193857 8946 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 09:01:36.194012 8931 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 09:01:36.194088 8931 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859658134510-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859658134510-392-0/minicluster-data/ts-2/wal
uuid: "5a350609debf460e84a8cdcbb37f0e5c"
format_stamp: "Formatted at 2026-06-19 09:01:36 on dist-test-slave-mtdr"
server_key: "226e48a95a242b4cbf174ddd2511e57e"
server_key_iv: "78d8a042da4fd84eed1cb29c653aef55"
server_key_version: "encryptionkey@0"
I20260619 09:01:36.194152 8931 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859658134510-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859658134510-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859658134510-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 09:01:36.201710 8931 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 09:01:36.203696 8931 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 09:01:36.203820 8931 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 09:01:36.204010 8931 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 09:01:36.204280 8931 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 09:01:36.204311 8931 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 09:01:36.204337 8931 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 09:01:36.204355 8931 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 09:01:36.209623 8931 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35399
I20260619 09:01:36.209668 9059 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35399 every 8 connection(s)
I20260619 09:01:36.209951 8931 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859658134510-392-0/minicluster-data/ts-2/data/info.pb
I20260619 09:01:36.213616 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ez5NB/build/release/bin/kudu as pid 8931
I20260619 09:01:36.213726 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859658134510-392-0/minicluster-data/ts-2/wal/instance
I20260619 09:01:36.213953 392 external_mini_cluster.cc:1428] Setting key 08446283700e0166953d67f70f3bcf54
Jun 19 09:01:36 dist-test-slave-mtdr krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859696, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 09:01:36.218719 8640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:36.210586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47199 (local address 127.0.98.62:34829)
0619 09:01:36.210640 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:36.210642 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:36.210854 (+ 212us) server_negotiation.cc:407] Connection header received
0619 09:01:36.211765 (+ 911us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:36.211767 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:36.211777 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:36.211835 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:01:36.212369 (+ 534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:36.212923 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:36.214142 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:36.214430 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:36.214492 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:36.216515 (+ 2023us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:01:36.216516 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:01:36.216516 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:01:36.216523 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:01:36.217723 (+ 1200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:01:36.218055 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:01:36.218055 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:01:36.218056 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:01:36.218104 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:01:36.218275 (+ 171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:01:36.218275 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:01:36.218276 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:01:36.218370 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:01:36.218414 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0619 09:01:36.218603 (+ 189us) server_negotiation.cc:299] Negotiation successful
0619 09:01:36.218656 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260619 09:01:36.218927 9062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:36.210563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34829 (local address 127.0.98.3:47199)
0619 09:01:36.210714 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0619 09:01:36.210755 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 09:01:36.211663 (+ 908us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:01:36.211894 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:01:36.211898 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:01:36.212120 (+ 222us) client_negotiation.cc:189] Negotiated authn=SASL
0619 09:01:36.212299 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:36.212303 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:36.213000 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:36.213001 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:36.214062 (+ 1061us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:36.214065 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:36.214509 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:36.214510 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:36.214629 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:36.215138 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 09:01:36.215148 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 09:01:36.216438 (+ 1290us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 09:01:36.217787 (+ 1349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:01:36.217792 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:01:36.217794 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:01:36.217993 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:01:36.218151 (+ 158us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:01:36.218152 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:01:36.218152 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:01:36.218235 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:01:36.218413 (+ 178us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 09:01:36.218415 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 09:01:36.218518 (+ 103us) client_negotiation.cc:769] Sending connection context
0619 09:01:36.218592 (+ 74us) client_negotiation.cc:240] Negotiation successful
0619 09:01:36.218701 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":70,"threads_started":1}
I20260619 09:01:36.219398 9060 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34829
I20260619 09:01:36.219473 9060 heartbeater.cc:464] Registering TS with master...
I20260619 09:01:36.219605 9060 heartbeater.cc:511] Master 127.0.98.62:34829 requested a full tablet report, sending...
I20260619 09:01:36.219892 8601 ts_manager.cc:194] Registered new tserver with Master: 5a350609debf460e84a8cdcbb37f0e5c (127.0.98.3:35399)
I20260619 09:01:36.220220 8601 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:47199
I20260619 09:01:36.225400 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859658134510-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859658134510-392-0/krb5kdc/oryx_127.0.98.1.keytab.
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859658134510-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859658134510-392-0/krb5kdc/oryx_127.0.98.2.keytab.
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859658134510-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859658134510-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260619 09:01:36.287659 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ez5NB/build/release/bin/kudu with pid 8663
I20260619 09:01:36.291759 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ez5NB/build/release/bin/kudu with pid 8797
I20260619 09:01:36.295202 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ez5NB/build/release/bin/kudu with pid 8931
I20260619 09:01:36.298760 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ez5NB/build/release/bin/kudu with pid 8569
I20260619 09:01:36.302937 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ez5NB/build/release/bin/kudu
/tmp/dist-test-task4ez5NB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859658134510-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859658134510-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859658134510-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859658134510-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34829
--webserver_interface=127.0.98.62
--webserver_port=45893
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859658134510-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34829
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859658134510-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859658134510-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859658134510-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 09:01:36.380084 9069 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 09:01:36.380311 9069 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 09:01:36.380335 9069 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 09:01:36.381433 9069 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 09:01:36.381465 9069 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 09:01:36.381477 9069 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 09:01:36.381489 9069 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 09:01:36.381498 9069 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 09:01:36.382936 9069 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859658134510-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859658134510-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34829
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859658134510-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34829
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859658134510-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=45893
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9069
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859658134510-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 09:01:36.383113 9069 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 09:01:36.383275 9069 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 09:01:36.387580 9075 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:01:36.387595 9074 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:01:36.387619 9077 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:01:36.387669 9069 server_base.cc:1034] running on GCE node
I20260619 09:01:36.388000 9069 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 09:01:36.388279 9069 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 09:01:36.389427 9069 hybrid_clock.cc:648] HybridClock initialized: now 1781859696389412 us; error 28 us; skew 500 ppm
Jun 19 09:01:36 dist-test-slave-mtdr krb5kdc[8556](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859696, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 09:01:36.391670 9069 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 09:01:36.392093 9069 webserver.cc:460] Webserver started at http://127.0.98.62:45893/ using document root <none> and password file <none>
I20260619 09:01:36.392282 9069 fs_manager.cc:359] Metadata directory not provided
I20260619 09:01:36.392341 9069 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 09:01:36.393585 9069 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 09:01:36.394137 9084 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 09:01:36.394261 9069 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 09:01:36.394340 9069 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859658134510-392-0/minicluster-data/master-0/data,/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859658134510-392-0/minicluster-data/master-0/wal
uuid: "22161c26406a4056857a369b1a64c600"
format_stamp: "Formatted at 2026-06-19 09:01:35 on dist-test-slave-mtdr"
server_key: "14570fa236eeec33262a5be827374ba1"
server_key_iv: "87460b10b23dcb9a3fbb437f52b8d98f"
server_key_version: "encryptionkey@0"
I20260619 09:01:36.394659 9069 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859658134510-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859658134510-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859658134510-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 09:01:36.416350 9069 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 09:01:36.418123 9069 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 09:01:36.418252 9069 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 09:01:36.421916 9069 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34829
I20260619 09:01:36.421972 9136 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34829 every 8 connection(s)
I20260619 09:01:36.422437 9069 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859658134510-392-0/minicluster-data/master-0/data/info.pb
I20260619 09:01:36.423681 9137 sys_catalog.cc:263] Verifying existing consensus state
I20260619 09:01:36.424299 9137 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 22161c26406a4056857a369b1a64c600: Bootstrap starting.
I20260619 09:01:36.427098 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ez5NB/build/release/bin/kudu as pid 9069
I20260619 09:01:36.427404 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ez5NB/build/release/bin/kudu
/tmp/dist-test-task4ez5NB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859658134510-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859658134510-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859658134510-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859658134510-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:35029
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:34829
--webserver_port=37433
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859658134510-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.1
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859658134510-392-0/krb5kdc/oryx_127.0.98.1.keytab with env {KRB5CCNAME=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859658134510-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859658134510-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859658134510-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 09:01:36.432349 9137 log.cc:824] T 00000000000000000000000000000000 P 22161c26406a4056857a369b1a64c600: Log is configured to *not* fsync() on all Append() calls
I20260619 09:01:36.433845 9137 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 22161c26406a4056857a369b1a64c600: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 09:01:36.434098 9137 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 22161c26406a4056857a369b1a64c600: Bootstrap complete.
I20260619 09:01:36.436123 9137 raft_consensus.cc:348] T 00000000000000000000000000000000 P 22161c26406a4056857a369b1a64c600 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22161c26406a4056857a369b1a64c600" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34829 } }
I20260619 09:01:36.436335 9137 raft_consensus.cc:729] T 00000000000000000000000000000000 P 22161c26406a4056857a369b1a64c600 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22161c26406a4056857a369b1a64c600, State: Initialized, Role: FOLLOWER
I20260619 09:01:36.436468 9137 consensus_queue.cc:260] T 00000000000000000000000000000000 P 22161c26406a4056857a369b1a64c600 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22161c26406a4056857a369b1a64c600" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34829 } }
I20260619 09:01:36.436532 9137 raft_consensus.cc:388] T 00000000000000000000000000000000 P 22161c26406a4056857a369b1a64c600 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 09:01:36.436573 9137 raft_consensus.cc:482] T 00000000000000000000000000000000 P 22161c26406a4056857a369b1a64c600 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 09:01:36.436615 9137 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 22161c26406a4056857a369b1a64c600 [term 1 FOLLOWER]: Advancing to term 2
I20260619 09:01:36.437578 9137 raft_consensus.cc:504] T 00000000000000000000000000000000 P 22161c26406a4056857a369b1a64c600 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22161c26406a4056857a369b1a64c600" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34829 } }
I20260619 09:01:36.437665 9137 leader_election.cc:302] T 00000000000000000000000000000000 P 22161c26406a4056857a369b1a64c600 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 22161c26406a4056857a369b1a64c600; no voters:
I20260619 09:01:36.437812 9137 leader_election.cc:288] T 00000000000000000000000000000000 P 22161c26406a4056857a369b1a64c600 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260619 09:01:36.437860 9141 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 22161c26406a4056857a369b1a64c600 [term 2 FOLLOWER]: Leader election won for term 2
I20260619 09:01:36.437963 9141 raft_consensus.cc:686] T 00000000000000000000000000000000 P 22161c26406a4056857a369b1a64c600 [term 2 LEADER]: Becoming Leader. State: Replica: 22161c26406a4056857a369b1a64c600, State: Running, Role: LEADER
I20260619 09:01:36.438045 9141 consensus_queue.cc:237] T 00000000000000000000000000000000 P 22161c26406a4056857a369b1a64c600 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22161c26406a4056857a369b1a64c600" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34829 } }
I20260619 09:01:36.438097 9137 sys_catalog.cc:564] T 00000000000000000000000000000000 P 22161c26406a4056857a369b1a64c600 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 09:01:36.438297 9141 sys_catalog.cc:455] T 00000000000000000000000000000000 P 22161c26406a4056857a369b1a64c600 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 22161c26406a4056857a369b1a64c600. Latest consensus state: current_term: 2 leader_uuid: "22161c26406a4056857a369b1a64c600" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22161c26406a4056857a369b1a64c600" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34829 } } }
I20260619 09:01:36.438360 9141 sys_catalog.cc:458] T 00000000000000000000000000000000 P 22161c26406a4056857a369b1a64c600 [sys.catalog]: This master's current role is: LEADER
I20260619 09:01:36.438498 9142 sys_catalog.cc:455] T 00000000000000000000000000000000 P 22161c26406a4056857a369b1a64c600 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "22161c26406a4056857a369b1a64c600" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22161c26406a4056857a369b1a64c600" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34829 } } }
I20260619 09:01:36.438556 9142 sys_catalog.cc:458] T 00000000000000000000000000000000 P 22161c26406a4056857a369b1a64c600 [sys.catalog]: This master's current role is: LEADER
I20260619 09:01:36.438776 9145 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 09:01:36.439298 9145 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 09:01:36.439419 9145 catalog_manager.cc:1259] Loaded cluster ID: 2a52b1c37c9c471da64ab3c9b6e20e89
I20260619 09:01:36.439658 9145 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 09:01:36.440871 9145 catalog_manager.cc:1504] Loading token signing keys...
I20260619 09:01:36.441039 9145 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 22161c26406a4056857a369b1a64c600: Loaded TSK: 0
I20260619 09:01:36.441242 9145 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 09:01:36.506968 9139 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 09:01:36.507157 9139 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 09:01:36.507179 9139 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 09:01:36.508319 9139 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 09:01:36.508359 9139 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 09:01:36.508383 9139 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 09:01:36.509759 9139 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859658134510-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859658134510-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859658134510-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:35029
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859658134510-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=37433
--tserver_master_addrs=127.0.98.62:34829
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9139
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859658134510-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 09:01:36.509958 9139 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 09:01:36.510169 9139 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 09:01:36.514763 9163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:01:36.514921 9139 server_base.cc:1034] running on GCE node
W20260619 09:01:36.514765 9164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:01:36.515096 9166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:01:36.515311 9139 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 09:01:36.515555 9139 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 09:01:36.516721 9139 hybrid_clock.cc:648] HybridClock initialized: now 1781859696516706 us; error 30 us; skew 500 ppm
Jun 19 09:01:36 dist-test-slave-mtdr krb5kdc[8556](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859696, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 09:01:36.519275 9139 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260619 09:01:36.519703 9139 webserver.cc:460] Webserver started at http://127.0.98.1:37433/ using document root <none> and password file <none>
I20260619 09:01:36.519912 9139 fs_manager.cc:359] Metadata directory not provided
I20260619 09:01:36.519976 9139 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 09:01:36.521234 9139 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 09:01:36.521775 9173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 09:01:36.521914 9139 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260619 09:01:36.521987 9139 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859658134510-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859658134510-392-0/minicluster-data/ts-0/wal
uuid: "557ed5c591424db0b4456ca89ef08d6a"
format_stamp: "Formatted at 2026-06-19 09:01:35 on dist-test-slave-mtdr"
server_key: "f7a4f9d4facf04911a81f14b547e39cb"
server_key_iv: "86b0d69ab74c67b323cb129532e0ff82"
server_key_version: "encryptionkey@0"
I20260619 09:01:36.522302 9139 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859658134510-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859658134510-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859658134510-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 09:01:36.532613 9139 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 09:01:36.534544 9139 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 09:01:36.534667 9139 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 09:01:36.534869 9139 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 09:01:36.535169 9139 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 09:01:36.535204 9139 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 09:01:36.535231 9139 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 09:01:36.535254 9139 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 09:01:36.541640 9139 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35029
I20260619 09:01:36.541672 9286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35029 every 8 connection(s)
I20260619 09:01:36.541983 9139 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859658134510-392-0/minicluster-data/ts-0/data/info.pb
I20260619 09:01:36.542152 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ez5NB/build/release/bin/kudu as pid 9139
I20260619 09:01:36.542449 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ez5NB/build/release/bin/kudu
/tmp/dist-test-task4ez5NB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859658134510-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859658134510-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859658134510-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859658134510-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:37743
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:34829
--webserver_port=37635
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859658134510-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.2
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859658134510-392-0/krb5kdc/oryx_127.0.98.2.keytab with env {KRB5CCNAME=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859658134510-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859658134510-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859658134510-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 09:01:36 dist-test-slave-mtdr krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 09:01:36 dist-test-slave-mtdr krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 09:01:36.552657 9289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:36.542724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34829 (local address 127.0.98.1:38843)
0619 09:01:36.542906 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0619 09:01:36.542947 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 09:01:36.543403 (+ 456us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:01:36.544450 (+ 1047us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:01:36.544455 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:01:36.545142 (+ 687us) client_negotiation.cc:189] Negotiated authn=SASL
0619 09:01:36.545391 (+ 249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:36.545397 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:36.547017 (+ 1620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:36.547018 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:36.548263 (+ 1245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:36.548265 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:36.549071 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:36.549072 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:36.549213 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:36.549791 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 09:01:36.549802 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 09:01:36.552456 (+ 2654us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34829: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":94,"threads_started":1}
W20260619 09:01:36.552875 9287 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:34829 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:34829: Client connection negotiation failed: client connection to 127.0.98.62:34829: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
W20260619 09:01:36.553035 9291 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 09:01:36.543824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38843 (local address 127.0.98.62:34829)
0619 09:01:36.544016 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:36.544025 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:36.544063 (+ 38us) server_negotiation.cc:407] Connection header received
0619 09:01:36.544186 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:36.544192 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:36.544245 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:36.544379 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:01:36.545524 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:36.546902 (+ 1378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:36.548353 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:36.548967 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:36.549082 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:36.552594 (+ 3512us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:38843: BlockingRecv error: recv got EOF from 127.0.98.1:38843 (error 108)
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":80,"threads_started":1}
Jun 19 09:01:36 dist-test-slave-mtdr krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 09:01:36 dist-test-slave-mtdr krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 09:01:36.558732 9289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:36.553076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34829 (local address 127.0.98.1:38143)
0619 09:01:36.553445 (+ 369us) negotiation.cc:107] Waiting for socket to connect
0619 09:01:36.553459 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 09:01:36.553521 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:01:36.553698 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:01:36.553699 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:01:36.553850 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0619 09:01:36.553904 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:36.553907 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:36.554845 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:36.554846 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:36.555679 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:36.555681 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:36.556127 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:36.556128 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:36.556192 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:36.556627 (+ 435us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 09:01:36.556632 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 09:01:36.558655 (+ 2023us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34829: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":83,"threads_started":1}
W20260619 09:01:36.559288 9291 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 09:01:36.553253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38143 (local address 127.0.98.62:34829)
0619 09:01:36.553405 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:36.553406 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:36.553517 (+ 111us) server_negotiation.cc:407] Connection header received
0619 09:01:36.553570 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:36.553571 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:36.553580 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:36.553665 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:01:36.553975 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:36.554735 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:36.555752 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:36.556064 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:36.556126 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:36.558749 (+ 2623us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:38143: BlockingRecv error: recv got EOF from 127.0.98.1:38143 (error 108)
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":76,"threads_started":1}
Jun 19 09:01:36 dist-test-slave-mtdr krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 09:01:36 dist-test-slave-mtdr krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 09:01:36.563752 9289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:36.558985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34829 (local address 127.0.98.1:58437)
0619 09:01:36.559025 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0619 09:01:36.559036 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 09:01:36.559076 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:01:36.559227 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:01:36.559228 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:01:36.559483 (+ 255us) client_negotiation.cc:189] Negotiated authn=SASL
0619 09:01:36.559517 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:36.559519 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:36.560354 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:36.560355 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:36.561144 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:36.561146 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:36.561539 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:36.561540 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:36.561591 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:36.561981 (+ 390us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 09:01:36.561985 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 09:01:36.563690 (+ 1705us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34829: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":24}
W20260619 09:01:36.563877 9287 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 09:01:36.563959 9293 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 09:01:36.559074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58437 (local address 127.0.98.62:34829)
0619 09:01:36.559116 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:36.559117 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:36.559124 (+ 7us) server_negotiation.cc:407] Connection header received
0619 09:01:36.559144 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:36.559145 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:36.559154 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:36.559219 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:01:36.559587 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:36.560279 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:36.561223 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:36.561472 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:36.561536 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:36.563774 (+ 2238us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:58437: BlockingRecv error: recv got EOF from 127.0.98.1:58437 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260619 09:01:36.623667 9290 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 09:01:36.623857 9290 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 09:01:36.623878 9290 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 09:01:36.625031 9290 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 09:01:36.625069 9290 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 09:01:36.625097 9290 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 09:01:36.626524 9290 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859658134510-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859658134510-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859658134510-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:37743
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859658134510-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=37635
--tserver_master_addrs=127.0.98.62:34829
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9290
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859658134510-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 09:01:36.626744 9290 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 09:01:36.626957 9290 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 09:01:36.631482 9299 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:01:36.631537 9290 server_base.cc:1034] running on GCE node
W20260619 09:01:36.631470 9301 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:01:36.631472 9298 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:01:36.631807 9290 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 09:01:36.632021 9290 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 09:01:36.633159 9290 hybrid_clock.cc:648] HybridClock initialized: now 1781859696633139 us; error 32 us; skew 500 ppm
Jun 19 09:01:36 dist-test-slave-mtdr krb5kdc[8556](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859696, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 09:01:36.635375 9290 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260619 09:01:36.635777 9290 webserver.cc:460] Webserver started at http://127.0.98.2:37635/ using document root <none> and password file <none>
I20260619 09:01:36.635960 9290 fs_manager.cc:359] Metadata directory not provided
I20260619 09:01:36.636018 9290 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 09:01:36.637240 9290 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 09:01:36.637727 9308 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 09:01:36.637880 9290 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260619 09:01:36.637954 9290 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859658134510-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859658134510-392-0/minicluster-data/ts-1/wal
uuid: "b1e5855d7c84416d977defe417c4ebf5"
format_stamp: "Formatted at 2026-06-19 09:01:36 on dist-test-slave-mtdr"
server_key: "dfa264e6c1611491b9d8e891f9eb911d"
server_key_iv: "cd722e0299be4dc706e3890f6844f4a4"
server_key_version: "encryptionkey@0"
I20260619 09:01:36.638290 9290 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859658134510-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859658134510-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859658134510-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 09:01:36.660254 9290 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 09:01:36.661931 9290 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 09:01:36.662060 9290 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 09:01:36.662271 9290 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 09:01:36.662544 9290 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 09:01:36.662577 9290 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 09:01:36.662606 9290 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 09:01:36.662637 9290 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 09:01:36.668007 9290 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37743
I20260619 09:01:36.668056 9421 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37743 every 8 connection(s)
I20260619 09:01:36.668356 9290 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859658134510-392-0/minicluster-data/ts-1/data/info.pb
Jun 19 09:01:36 dist-test-slave-mtdr krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 09:01:36 dist-test-slave-mtdr krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260619 09:01:36.675104 9293 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 09:01:36.668975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45801 (local address 127.0.98.62:34829)
0619 09:01:36.669024 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:36.669026 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:36.669248 (+ 222us) server_negotiation.cc:407] Connection header received
0619 09:01:36.669748 (+ 500us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:36.669750 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:36.669763 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:36.669810 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:01:36.670739 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:36.671283 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:36.672235 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:36.672436 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:36.672490 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:36.675039 (+ 2549us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:45801: BlockingRecv error: recv got EOF from 127.0.98.2:45801 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
I20260619 09:01:36.675143 9424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:36.668983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34829 (local address 127.0.98.2:45801)
0619 09:01:36.669131 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0619 09:01:36.669166 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0619 09:01:36.669650 (+ 484us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:01:36.669889 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:01:36.669892 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:01:36.670443 (+ 551us) client_negotiation.cc:189] Negotiated authn=SASL
0619 09:01:36.670637 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:36.670642 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:36.671364 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:36.671365 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:36.672157 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:36.672159 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:36.672500 (+ 341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:36.672501 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:36.672620 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:36.673029 (+ 409us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 09:01:36.673036 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0619 09:01:36.674957 (+ 1921us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34829: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":73,"threads_started":1}
W20260619 09:01:36.675325 9422 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:34829 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:34829: Client connection negotiation failed: client connection to 127.0.98.62:34829: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
I20260619 09:01:36.677248 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ez5NB/build/release/bin/kudu as pid 9290
I20260619 09:01:36.677522 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ez5NB/build/release/bin/kudu
/tmp/dist-test-task4ez5NB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859658134510-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859658134510-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859658134510-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859658134510-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:35399
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:34829
--webserver_port=38665
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859658134510-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.3
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859658134510-392-0/krb5kdc/oryx_127.0.98.3.keytab with env {KRB5CCNAME=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859658134510-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859658134510-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859658134510-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 09:01:36 dist-test-slave-mtdr krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 09:01:36 dist-test-slave-mtdr krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 09:01:36.679461 9424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:36.675504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34829 (local address 127.0.98.2:36275)
0619 09:01:36.675602 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0619 09:01:36.675612 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 09:01:36.675655 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:01:36.675761 (+ 106us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:01:36.675762 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:01:36.675873 (+ 111us) client_negotiation.cc:189] Negotiated authn=SASL
0619 09:01:36.675911 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:36.675912 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:36.676483 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:36.676484 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:36.676975 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:36.676976 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:36.677364 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:36.677365 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:36.677423 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:36.677786 (+ 363us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 09:01:36.677790 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 09:01:36.679398 (+ 1608us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34829: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":84}
W20260619 09:01:36.679512 9293 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 09:01:36.675565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36275 (local address 127.0.98.62:34829)
0619 09:01:36.675607 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:36.675608 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:36.675642 (+ 34us) server_negotiation.cc:407] Connection header received
0619 09:01:36.675707 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:36.675708 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:36.675717 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:36.675760 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:01:36.675969 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:36.676422 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:36.677038 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:36.677298 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:36.677361 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:36.679471 (+ 2110us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:36275: BlockingRecv error: recv got EOF from 127.0.98.2:36275 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
Jun 19 09:01:36 dist-test-slave-mtdr krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 09:01:36 dist-test-slave-mtdr krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 09:01:36.683699 9424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:36.679783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34829 (local address 127.0.98.2:44711)
0619 09:01:36.679823 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0619 09:01:36.679833 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 09:01:36.679863 (+ 30us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:01:36.679995 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:01:36.679996 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:01:36.680120 (+ 124us) client_negotiation.cc:189] Negotiated authn=SASL
0619 09:01:36.680143 (+ 23us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:36.680144 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:36.680716 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:36.680717 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:36.681246 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:36.681248 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:36.681691 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:36.681692 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:36.681723 (+ 31us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:36.682043 (+ 320us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 09:01:36.682047 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 09:01:36.683657 (+ 1610us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34829: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":26}
W20260619 09:01:36.683799 9293 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 09:01:36.679844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44711 (local address 127.0.98.62:34829)
0619 09:01:36.679902 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:36.679904 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:36.679911 (+ 7us) server_negotiation.cc:407] Connection header received
0619 09:01:36.679928 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:36.679929 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:36.679938 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:36.679978 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:01:36.680201 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:36.680639 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:36.681325 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:36.681621 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:36.681697 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:36.683750 (+ 2053us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:44711: BlockingRecv error: recv got EOF from 127.0.98.2:44711 (error 108)
Metrics: {"server-negotiator.queue_time_us":28}
W20260619 09:01:36.683799 9422 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 09:01:36.755159 9425 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 09:01:36.755355 9425 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 09:01:36.755378 9425 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 09:01:36.756475 9425 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 09:01:36.756507 9425 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 09:01:36.756531 9425 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 09:01:36.757875 9425 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859658134510-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859658134510-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859658134510-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:35399
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859658134510-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=38665
--tserver_master_addrs=127.0.98.62:34829
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9425
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859658134510-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 09:01:36.758044 9425 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 09:01:36.758213 9425 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 09:01:36.762797 9433 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:01:36.762840 9431 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:01:36.762964 9425 server_base.cc:1034] running on GCE node
W20260619 09:01:36.762900 9430 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:01:36.763239 9425 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 09:01:36.763458 9425 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 09:01:36.764591 9425 hybrid_clock.cc:648] HybridClock initialized: now 1781859696764579 us; error 26 us; skew 500 ppm
Jun 19 09:01:36 dist-test-slave-mtdr krb5kdc[8556](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859696, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 09:01:36.766772 9425 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260619 09:01:36.767185 9425 webserver.cc:460] Webserver started at http://127.0.98.3:38665/ using document root <none> and password file <none>
I20260619 09:01:36.767377 9425 fs_manager.cc:359] Metadata directory not provided
I20260619 09:01:36.767431 9425 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 09:01:36.768663 9425 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260619 09:01:36.769233 9440 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 09:01:36.769382 9425 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 09:01:36.769455 9425 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859658134510-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859658134510-392-0/minicluster-data/ts-2/wal
uuid: "5a350609debf460e84a8cdcbb37f0e5c"
format_stamp: "Formatted at 2026-06-19 09:01:36 on dist-test-slave-mtdr"
server_key: "226e48a95a242b4cbf174ddd2511e57e"
server_key_iv: "78d8a042da4fd84eed1cb29c653aef55"
server_key_version: "encryptionkey@0"
I20260619 09:01:36.769768 9425 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859658134510-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859658134510-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859658134510-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 09:01:36.807574 9425 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 09:01:36.810079 9425 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 09:01:36.810240 9425 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 09:01:36.810494 9425 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 09:01:36.810878 9425 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 09:01:36.810922 9425 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 09:01:36.810956 9425 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 09:01:36.810974 9425 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 09:01:36.818442 9425 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35399
I20260619 09:01:36.818491 9553 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35399 every 8 connection(s)
I20260619 09:01:36.818913 9425 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781859658134510-392-0/minicluster-data/ts-2/data/info.pb
I20260619 09:01:36.822710 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ez5NB/build/release/bin/kudu as pid 9425
Jun 19 09:01:36 dist-test-slave-mtdr krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 09:01:36 dist-test-slave-mtdr krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260619 09:01:36.827672 9293 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 09:01:36.819596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42729 (local address 127.0.98.62:34829)
0619 09:01:36.819650 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:36.819652 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:36.819871 (+ 219us) server_negotiation.cc:407] Connection header received
0619 09:01:36.820378 (+ 507us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:36.820379 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:36.820390 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:36.820441 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:01:36.821392 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:36.822117 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:36.823290 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:36.823530 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:36.823567 (+ 37us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:36.827599 (+ 4032us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:42729: BlockingRecv error: recv got EOF from 127.0.98.3:42729 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
I20260619 09:01:36.827731 9556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:36.819614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34829 (local address 127.0.98.3:42729)
0619 09:01:36.819758 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0619 09:01:36.819798 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 09:01:36.820275 (+ 477us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:01:36.820511 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:01:36.820515 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:01:36.821090 (+ 575us) client_negotiation.cc:189] Negotiated authn=SASL
0619 09:01:36.821294 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:36.821301 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:36.822210 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:36.822212 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:36.823193 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:36.823196 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:36.823703 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:36.823704 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:36.823840 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:36.824993 (+ 1153us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 09:01:36.825003 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 09:01:36.827502 (+ 2499us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34829: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":77,"threads_started":1}
W20260619 09:01:36.827950 9554 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:34829 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:34829: Client connection negotiation failed: client connection to 127.0.98.62:34829: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
I20260619 09:01:36.831995 9291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:36.823616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40998 (local address 127.0.98.62:34829)
0619 09:01:36.823672 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:36.823673 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:36.823680 (+ 7us) server_negotiation.cc:407] Connection header received
0619 09:01:36.823697 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:36.823698 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:36.823708 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:36.823758 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:01:36.824647 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:36.825289 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:36.826362 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:36.826606 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:36.826672 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:36.828297 (+ 1625us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:01:36.828302 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:01:36.828303 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:01:36.828315 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:01:36.830225 (+ 1910us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:01:36.830912 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:01:36.830915 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:01:36.830916 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:01:36.831037 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:01:36.831253 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:01:36.831254 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:01:36.831254 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:01:36.831513 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:01:36.831665 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0619 09:01:36.831791 (+ 126us) server_negotiation.cc:299] Negotiation successful
0619 09:01:36.831902 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
Jun 19 09:01:36 dist-test-slave-mtdr krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 09:01:36 dist-test-slave-mtdr krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260619 09:01:36.833695 9293 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 09:01:36.828218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34897 (local address 127.0.98.62:34829)
0619 09:01:36.828253 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:36.828254 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:36.828625 (+ 371us) server_negotiation.cc:407] Connection header received
0619 09:01:36.828703 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:36.828704 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:36.828713 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:36.828758 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:01:36.829060 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:36.829724 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:36.830562 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:36.830796 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:36.830873 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:36.833628 (+ 2755us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:34897: BlockingRecv error: recv got EOF from 127.0.98.3:34897 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
I20260619 09:01:36.833859 9556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:36.828169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34829 (local address 127.0.98.3:34897)
0619 09:01:36.828568 (+ 399us) negotiation.cc:107] Waiting for socket to connect
0619 09:01:36.828577 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 09:01:36.828621 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:01:36.828793 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:01:36.828794 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:01:36.828925 (+ 131us) client_negotiation.cc:189] Negotiated authn=SASL
0619 09:01:36.828975 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:36.828977 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:36.829814 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:36.829815 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:36.830499 (+ 684us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:36.830501 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:36.830923 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:36.830924 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:36.830986 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:36.831497 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 09:01:36.831502 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 09:01:36.833802 (+ 2300us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34829: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":384}
Jun 19 09:01:36 dist-test-slave-mtdr krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 09:01:36 dist-test-slave-mtdr krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 09:01:36.837647 9556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:36.834080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34829 (local address 127.0.98.3:32779)
0619 09:01:36.834110 (+ 30us) negotiation.cc:107] Waiting for socket to connect
0619 09:01:36.834120 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 09:01:36.834164 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:01:36.834283 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:01:36.834284 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:01:36.834393 (+ 109us) client_negotiation.cc:189] Negotiated authn=SASL
0619 09:01:36.834418 (+ 25us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:36.834419 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:36.835057 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:36.835058 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:36.835536 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:36.835538 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:36.835856 (+ 318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:36.835857 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:36.835904 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:36.836222 (+ 318us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 09:01:36.836226 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 09:01:36.837585 (+ 1359us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34829: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":15}
W20260619 09:01:36.837706 9293 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 09:01:36.834167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:32779 (local address 127.0.98.62:34829)
0619 09:01:36.834205 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:36.834207 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:36.834213 (+ 6us) server_negotiation.cc:407] Connection header received
0619 09:01:36.834229 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:36.834230 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:36.834238 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:36.834279 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:01:36.834477 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:36.834987 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:36.835599 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:36.835800 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:36.835852 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:36.837659 (+ 1807us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:32779: BlockingRecv error: recv got EOF from 127.0.98.3:32779 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260619 09:01:36.837819 9554 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Jun 19 09:01:37 dist-test-slave-mtdr krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 09:01:37 dist-test-slave-mtdr krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 09:01:37.568424 9558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:37.564243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34829 (local address 127.0.98.1:45441)
0619 09:01:37.564366 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0619 09:01:37.564376 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 09:01:37.564428 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:01:37.564552 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:01:37.564552 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:01:37.564696 (+ 144us) client_negotiation.cc:189] Negotiated authn=SASL
0619 09:01:37.564745 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:37.564746 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:37.565342 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:37.565343 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:37.565863 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:37.565864 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:37.566272 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:37.566273 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:37.566338 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:37.566781 (+ 443us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 09:01:37.566785 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 09:01:37.568359 (+ 1574us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34829: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":67,"threads_started":1}
W20260619 09:01:37.568513 9559 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 09:01:37.564321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45441 (local address 127.0.98.62:34829)
0619 09:01:37.564429 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:37.564431 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:37.564457 (+ 26us) server_negotiation.cc:407] Connection header received
0619 09:01:37.564477 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:37.564477 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:37.564486 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:37.564541 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:01:37.564796 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:37.565290 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:37.565947 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:37.566198 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:37.566269 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:37.568447 (+ 2178us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:45441: BlockingRecv error: recv got EOF from 127.0.98.1:45441 (error 108)
Metrics: {"server-negotiator.queue_time_us":89,"thread_start_us":53,"threads_started":1}
Jun 19 09:01:37 dist-test-slave-mtdr krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 09:01:37 dist-test-slave-mtdr krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 09:01:37.688256 9560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:37.684180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34829 (local address 127.0.98.2:38675)
0619 09:01:37.684320 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0619 09:01:37.684330 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 09:01:37.684381 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:01:37.684488 (+ 107us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:01:37.684488 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:01:37.684637 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0619 09:01:37.684684 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:37.684685 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:37.685282 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:37.685282 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:37.685816 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:37.685818 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:37.686117 (+ 299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:37.686117 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:37.686161 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:37.686593 (+ 432us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 09:01:37.686598 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 09:01:37.688189 (+ 1591us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34829: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":62,"threads_started":1}
W20260619 09:01:37.688364 9559 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 09:01:37.684225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38675 (local address 127.0.98.62:34829)
0619 09:01:37.684267 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:37.684269 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:37.684368 (+ 99us) server_negotiation.cc:407] Connection header received
0619 09:01:37.684425 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:37.684425 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:37.684435 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:37.684472 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:01:37.684733 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:37.685219 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:37.685878 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:37.686055 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:37.686102 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:37.688304 (+ 2202us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:38675: BlockingRecv error: recv got EOF from 127.0.98.2:38675 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
Jun 19 09:01:37 dist-test-slave-mtdr krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 09:01:37 dist-test-slave-mtdr krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 09:01:37.842796 9561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:37.838162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34829 (local address 127.0.98.3:58547)
0619 09:01:37.838307 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0619 09:01:37.838322 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 09:01:37.838399 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:01:37.838507 (+ 108us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:01:37.838507 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:01:37.838655 (+ 148us) client_negotiation.cc:189] Negotiated authn=SASL
0619 09:01:37.838700 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:37.838702 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:37.839399 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:37.839400 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:37.839946 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:37.839948 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:37.840328 (+ 380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:37.840329 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:37.840373 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:37.840897 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 09:01:37.840903 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0619 09:01:37.842711 (+ 1808us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34829: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":68,"threads_started":1}
W20260619 09:01:37.842922 9559 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 09:01:37.838220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58547 (local address 127.0.98.62:34829)
0619 09:01:37.838274 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:37.838276 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:37.838382 (+ 106us) server_negotiation.cc:407] Connection header received
0619 09:01:37.838442 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:37.838443 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:37.838454 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:37.838501 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:01:37.838754 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:37.839310 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:37.840036 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:37.840258 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:37.840315 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:37.842849 (+ 2534us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:58547: BlockingRecv error: recv got EOF from 127.0.98.3:58547 (error 108)
Metrics: {"server-negotiator.queue_time_us":28}
Jun 19 09:01:38 dist-test-slave-mtdr krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 09:01:38 dist-test-slave-mtdr krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 09:01:38.573009 9562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:38.568841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34829 (local address 127.0.98.1:46521)
0619 09:01:38.568967 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0619 09:01:38.568978 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 09:01:38.569028 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:01:38.569148 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:01:38.569148 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:01:38.569319 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0619 09:01:38.569368 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:38.569369 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:38.569976 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:38.569977 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:38.570503 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:38.570504 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:38.570854 (+ 350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:38.570856 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:38.570915 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:38.571347 (+ 432us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 09:01:38.571351 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 09:01:38.572910 (+ 1559us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34829: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":66,"threads_started":1}
W20260619 09:01:38.573081 9563 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 09:01:38.568920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46521 (local address 127.0.98.62:34829)
0619 09:01:38.569029 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:38.569031 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:38.569060 (+ 29us) server_negotiation.cc:407] Connection header received
0619 09:01:38.569079 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:38.569079 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:38.569088 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:38.569136 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:01:38.569424 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:38.569923 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:38.570562 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:38.570772 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:38.570865 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:38.573002 (+ 2137us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:46521: BlockingRecv error: recv got EOF from 127.0.98.1:46521 (error 108)
Metrics: {"server-negotiator.queue_time_us":90,"thread_start_us":53,"threads_started":1}
Jun 19 09:01:38 dist-test-slave-mtdr krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 09:01:38 dist-test-slave-mtdr krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 09:01:38.693145 9564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:38.688738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34829 (local address 127.0.98.2:52205)
0619 09:01:38.688865 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0619 09:01:38.688876 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 09:01:38.688933 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:01:38.689061 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:01:38.689062 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:01:38.689213 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0619 09:01:38.689258 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:38.689260 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:38.689941 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:38.689942 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:38.690492 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:38.690494 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:38.690903 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:38.690904 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:38.690956 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:38.691450 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 09:01:38.691455 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 09:01:38.693064 (+ 1609us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34829: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":60,"threads_started":1}
W20260619 09:01:38.693218 9563 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 09:01:38.688806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52205 (local address 127.0.98.62:34829)
0619 09:01:38.688890 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:38.688892 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:38.688918 (+ 26us) server_negotiation.cc:407] Connection header received
0619 09:01:38.688985 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:38.688986 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:38.688999 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:38.689052 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:01:38.689343 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:38.689840 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:38.690587 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:38.690783 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:38.690884 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:38.693165 (+ 2281us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:52205: BlockingRecv error: recv got EOF from 127.0.98.2:52205 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Jun 19 09:01:38 dist-test-slave-mtdr krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 09:01:38 dist-test-slave-mtdr krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 09:01:38.847501 9565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:38.843265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34829 (local address 127.0.98.3:43419)
0619 09:01:38.843409 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0619 09:01:38.843421 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 09:01:38.843491 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:01:38.843618 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:01:38.843619 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:01:38.843780 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0619 09:01:38.843824 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:38.843826 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:38.844464 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:38.844466 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:38.844984 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:38.844986 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:38.845305 (+ 319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:38.845306 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:38.845358 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:38.845779 (+ 421us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 09:01:38.845785 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0619 09:01:38.847422 (+ 1637us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34829: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":67,"threads_started":1}
W20260619 09:01:38.847572 9563 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 09:01:38.843379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43419 (local address 127.0.98.62:34829)
0619 09:01:38.843428 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:38.843430 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:38.843474 (+ 44us) server_negotiation.cc:407] Connection header received
0619 09:01:38.843552 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:38.843552 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:38.843565 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:38.843603 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:01:38.843893 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:38.844388 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:38.845057 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:38.845249 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:38.845302 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:38.847516 (+ 2214us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:43419: BlockingRecv error: recv got EOF from 127.0.98.3:43419 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
Jun 19 09:01:39 dist-test-slave-mtdr krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 09:01:39 dist-test-slave-mtdr krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 09:01:39.578025 9566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:39.573451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34829 (local address 127.0.98.1:48247)
0619 09:01:39.573633 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0619 09:01:39.573645 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 09:01:39.573707 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:01:39.573850 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:01:39.573851 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:01:39.574007 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0619 09:01:39.574055 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:39.574057 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:39.574664 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:39.574665 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:39.575277 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:39.575280 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:39.575620 (+ 340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:39.575621 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:39.575670 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:39.576189 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 09:01:39.576195 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0619 09:01:39.577959 (+ 1764us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34829: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":60,"threads_started":1}
W20260619 09:01:39.578148 9567 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 09:01:39.573481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48247 (local address 127.0.98.62:34829)
0619 09:01:39.573607 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:39.573609 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:39.573708 (+ 99us) server_negotiation.cc:407] Connection header received
0619 09:01:39.573761 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:39.573762 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:39.573774 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:39.573853 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:01:39.574111 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:39.574596 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:39.575345 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:39.575533 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:39.575584 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:39.578073 (+ 2489us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:48247: BlockingRecv error: recv got EOF from 127.0.98.1:48247 (error 108)
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":65,"threads_started":1}
Jun 19 09:01:39 dist-test-slave-mtdr krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 09:01:39 dist-test-slave-mtdr krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 09:01:39.698010 9568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:39.693681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34829 (local address 127.0.98.2:56995)
0619 09:01:39.693814 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0619 09:01:39.693824 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 09:01:39.693878 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:01:39.694007 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:01:39.694008 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:01:39.694157 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0619 09:01:39.694203 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:39.694204 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:39.694890 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:39.694891 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:39.695421 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:39.695423 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:39.695719 (+ 296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:39.695720 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:39.695757 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:39.696195 (+ 438us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 09:01:39.696201 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0619 09:01:39.697921 (+ 1720us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34829: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":78,"threads_started":1}
W20260619 09:01:39.698086 9567 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 09:01:39.693789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56995 (local address 127.0.98.62:34829)
0619 09:01:39.693855 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:39.693857 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:39.693865 (+ 8us) server_negotiation.cc:407] Connection header received
0619 09:01:39.693936 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:39.693936 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:39.693946 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:39.693995 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:01:39.694263 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:39.694761 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:39.695485 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:39.695664 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:39.695715 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:39.698028 (+ 2313us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:56995: BlockingRecv error: recv got EOF from 127.0.98.2:56995 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Jun 19 09:01:39 dist-test-slave-mtdr krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 09:01:39 dist-test-slave-mtdr krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 09:01:39.852072 9569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:39.847946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34829 (local address 127.0.98.3:51121)
0619 09:01:39.848073 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0619 09:01:39.848086 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 09:01:39.848163 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:01:39.848295 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:01:39.848296 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:01:39.848451 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0619 09:01:39.848494 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:39.848496 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:39.849125 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:39.849127 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:39.849642 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:39.849643 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:39.849952 (+ 309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:39.849953 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:39.850005 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:39.850425 (+ 420us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 09:01:39.850429 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 09:01:39.851977 (+ 1548us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34829: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":62,"threads_started":1}
W20260619 09:01:39.852118 9567 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 09:01:39.848046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51121 (local address 127.0.98.62:34829)
0619 09:01:39.848118 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:39.848119 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:39.848144 (+ 25us) server_negotiation.cc:407] Connection header received
0619 09:01:39.848218 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:39.848219 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:39.848232 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:39.848273 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:01:39.848572 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:39.849053 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:39.849715 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:39.849896 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:39.849948 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:39.852064 (+ 2116us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:51121: BlockingRecv error: recv got EOF from 127.0.98.3:51121 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Jun 19 09:01:40 dist-test-slave-mtdr krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 09:01:40 dist-test-slave-mtdr krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 09:01:40.582976 9570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:40.578564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34829 (local address 127.0.98.1:52091)
0619 09:01:40.578708 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0619 09:01:40.578718 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 09:01:40.578784 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:01:40.578948 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:01:40.578949 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:01:40.579098 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0619 09:01:40.579145 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:40.579147 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:40.579783 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:40.579784 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:40.580321 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:40.580322 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:40.580714 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:40.580715 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:40.580770 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:40.581244 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 09:01:40.581250 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0619 09:01:40.582908 (+ 1658us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34829: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":70,"threads_started":1}
W20260619 09:01:40.583055 9571 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 09:01:40.578695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52091 (local address 127.0.98.62:34829)
0619 09:01:40.578807 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:40.578809 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:40.578838 (+ 29us) server_negotiation.cc:407] Connection header received
0619 09:01:40.578860 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:40.578861 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:40.578871 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:40.578930 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:01:40.579206 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:40.579715 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:40.580413 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:40.580640 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:40.580713 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:40.582987 (+ 2274us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:52091: BlockingRecv error: recv got EOF from 127.0.98.1:52091 (error 108)
Metrics: {"server-negotiator.queue_time_us":89,"thread_start_us":51,"threads_started":1}
Jun 19 09:01:40 dist-test-slave-mtdr krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 09:01:40 dist-test-slave-mtdr krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 09:01:40.703107 9572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:40.698612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34829 (local address 127.0.98.2:53499)
0619 09:01:40.698746 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0619 09:01:40.698756 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 09:01:40.698830 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:01:40.698997 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:01:40.698998 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:01:40.699190 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0619 09:01:40.699275 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:40.699277 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:40.699969 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:40.699970 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:40.700523 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:40.700525 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:40.700846 (+ 321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:40.700847 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:40.700892 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:40.701337 (+ 445us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 09:01:40.701342 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 09:01:40.703039 (+ 1697us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34829: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":77,"threads_started":1}
W20260619 09:01:40.703222 9571 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 09:01:40.698687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53499 (local address 127.0.98.62:34829)
0619 09:01:40.698763 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:40.698765 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:40.698802 (+ 37us) server_negotiation.cc:407] Connection header received
0619 09:01:40.698911 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:40.698911 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:40.698921 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:40.698983 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:01:40.699350 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:40.699896 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:40.700606 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:40.700791 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:40.700844 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:40.703159 (+ 2315us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:53499: BlockingRecv error: recv got EOF from 127.0.98.2:53499 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Jun 19 09:01:40 dist-test-slave-mtdr krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 09:01:40 dist-test-slave-mtdr krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 09:01:40.859649 9573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:40.852582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34829 (local address 127.0.98.3:43911)
0619 09:01:40.852720 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0619 09:01:40.852730 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 09:01:40.852787 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:01:40.852902 (+ 115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:01:40.852902 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:01:40.853062 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0619 09:01:40.853112 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:40.853114 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:40.853793 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:40.853794 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:40.854320 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:40.854321 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:40.854662 (+ 341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:40.854663 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:40.854702 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:40.857897 (+ 3195us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 09:01:40.857903 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0619 09:01:40.859590 (+ 1687us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34829: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":69,"threads_started":1}
W20260619 09:01:40.859789 9571 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 09:01:40.852661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43911 (local address 127.0.98.62:34829)
0619 09:01:40.852714 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:40.852715 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:40.852782 (+ 67us) server_negotiation.cc:407] Connection header received
0619 09:01:40.852838 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:40.852839 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:40.852849 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:40.852891 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:01:40.853195 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:40.853703 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:40.854400 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:40.854601 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:40.854656 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:40.859717 (+ 5061us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:43911: BlockingRecv error: recv got EOF from 127.0.98.3:43911 (error 108)
Metrics: {"server-negotiator.queue_time_us":29}
Jun 19 09:01:41 dist-test-slave-mtdr krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 09:01:41 dist-test-slave-mtdr krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 09:01:41.587636 9574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:41.583368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34829 (local address 127.0.98.1:59055)
0619 09:01:41.583553 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0619 09:01:41.583567 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 09:01:41.583630 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:01:41.583753 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:01:41.583753 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:01:41.583902 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0619 09:01:41.583949 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:41.583951 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:41.584591 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:41.584592 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:41.585139 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:41.585140 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:41.585459 (+ 319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:41.585460 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:41.585513 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:41.585945 (+ 432us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 09:01:41.585950 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 09:01:41.587579 (+ 1629us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34829: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":66,"threads_started":1}
W20260619 09:01:41.587777 9575 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 09:01:41.583457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59055 (local address 127.0.98.62:34829)
0619 09:01:41.583592 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:41.583593 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:41.583616 (+ 23us) server_negotiation.cc:407] Connection header received
0619 09:01:41.583683 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:41.583684 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:41.583694 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:41.583740 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:01:41.584007 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:41.584516 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:41.585205 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:41.585398 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:41.585445 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:41.587693 (+ 2248us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:59055: BlockingRecv error: recv got EOF from 127.0.98.1:59055 (error 108)
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":78,"threads_started":1}
Jun 19 09:01:41 dist-test-slave-mtdr krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 09:01:41 dist-test-slave-mtdr krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 09:01:41.708068 9576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:41.703615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34829 (local address 127.0.98.2:44395)
0619 09:01:41.703758 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0619 09:01:41.703771 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 09:01:41.703837 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:01:41.703999 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:01:41.704000 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:01:41.704188 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0619 09:01:41.704237 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:41.704239 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:41.704903 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:41.704905 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:41.705488 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:41.705491 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:41.705845 (+ 354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:41.705846 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:41.705901 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:41.706337 (+ 436us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 09:01:41.706342 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 09:01:41.707963 (+ 1621us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34829: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":65,"threads_started":1}
W20260619 09:01:41.708125 9575 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 09:01:41.703729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44395 (local address 127.0.98.62:34829)
0619 09:01:41.703783 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:41.703786 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:41.703818 (+ 32us) server_negotiation.cc:407] Connection header received
0619 09:01:41.703918 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:41.703918 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:41.703931 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:41.703988 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:01:41.704316 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:41.704822 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:41.705577 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:41.705772 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:41.705828 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:41.708073 (+ 2245us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:44395: BlockingRecv error: recv got EOF from 127.0.98.2:44395 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
Jun 19 09:01:41 dist-test-slave-mtdr krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 09:01:41 dist-test-slave-mtdr krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 09:01:41.866065 9577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:41.860057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34829 (local address 127.0.98.3:59807)
0619 09:01:41.860197 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0619 09:01:41.860210 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 09:01:41.860282 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:01:41.860426 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:01:41.860427 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:01:41.860592 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0619 09:01:41.860641 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:41.860643 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:41.861635 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:41.861636 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:41.862491 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:41.862494 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:41.863052 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:41.863053 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:41.863119 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:41.863573 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 09:01:41.863579 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0619 09:01:41.865979 (+ 2400us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34829: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":58,"threads_started":1}
W20260619 09:01:41.866317 9575 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 09:01:41.860195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59807 (local address 127.0.98.62:34829)
0619 09:01:41.860250 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:41.860251 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:41.860262 (+ 11us) server_negotiation.cc:407] Connection header received
0619 09:01:41.860343 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:41.860344 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:41.860357 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:41.860410 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:01:41.860715 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:41.861553 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:41.862581 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:41.862984 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:41.863042 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:41.866248 (+ 3206us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:59807: BlockingRecv error: recv got EOF from 127.0.98.3:59807 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
Jun 19 09:01:42 dist-test-slave-mtdr krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 09:01:42 dist-test-slave-mtdr krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 09:01:42.592617 9578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:42.588100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34829 (local address 127.0.98.1:37153)
0619 09:01:42.588238 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0619 09:01:42.588249 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 09:01:42.588306 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:01:42.588430 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:01:42.588431 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:01:42.588608 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0619 09:01:42.588655 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:42.588657 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:42.589309 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:42.589310 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:42.589859 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:42.589860 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:42.590262 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:42.590263 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:42.590305 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:42.590775 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 09:01:42.590780 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 09:01:42.592527 (+ 1747us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34829: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":77,"threads_started":1}
W20260619 09:01:42.592701 9579 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 09:01:42.588173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37153 (local address 127.0.98.62:34829)
0619 09:01:42.588283 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:42.588285 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:42.588295 (+ 10us) server_negotiation.cc:407] Connection header received
0619 09:01:42.588355 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:42.588356 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:42.588365 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:42.588420 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:01:42.588709 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:42.589235 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:42.589932 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:42.590198 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:42.590252 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:42.592602 (+ 2350us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:37153: BlockingRecv error: recv got EOF from 127.0.98.1:37153 (error 108)
Metrics: {"server-negotiator.queue_time_us":88,"thread_start_us":51,"threads_started":1}
Jun 19 09:01:42 dist-test-slave-mtdr krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 09:01:42 dist-test-slave-mtdr krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 09:01:42.713068 9580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:42.708595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34829 (local address 127.0.98.2:36137)
0619 09:01:42.708752 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0619 09:01:42.708765 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 09:01:42.708826 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:01:42.708948 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:01:42.708949 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:01:42.709097 (+ 148us) client_negotiation.cc:189] Negotiated authn=SASL
0619 09:01:42.709148 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:42.709150 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:42.709769 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:42.709770 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:42.710311 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:42.710314 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:42.710672 (+ 358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:42.710673 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:42.710713 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:42.711264 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 09:01:42.711270 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0619 09:01:42.712990 (+ 1720us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34829: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":57,"threads_started":1}
W20260619 09:01:42.713140 9579 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 09:01:42.708779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36137 (local address 127.0.98.62:34829)
0619 09:01:42.708836 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:42.708839 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:42.708861 (+ 22us) server_negotiation.cc:407] Connection header received
0619 09:01:42.708884 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:42.708885 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:42.708895 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:42.708934 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:01:42.709206 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:42.709683 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:42.710405 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:42.710607 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:42.710672 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:42.713086 (+ 2414us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:36137: BlockingRecv error: recv got EOF from 127.0.98.2:36137 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
Jun 19 09:01:42 dist-test-slave-mtdr krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 09:01:42 dist-test-slave-mtdr krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 09:01:42.871032 9581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:42.866590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34829 (local address 127.0.98.3:43691)
0619 09:01:42.866725 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0619 09:01:42.866738 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 09:01:42.866842 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:01:42.867035 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:01:42.867036 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:01:42.867200 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0619 09:01:42.867247 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:42.867248 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:42.867886 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:42.867887 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:42.868464 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:42.868466 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:42.868806 (+ 340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:42.868808 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:42.868853 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:42.869275 (+ 422us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 09:01:42.869279 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 09:01:42.870963 (+ 1684us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34829: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":70,"threads_started":1}
W20260619 09:01:42.871088 9579 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 09:01:42.866701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43691 (local address 127.0.98.62:34829)
0619 09:01:42.866773 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:42.866774 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:42.866783 (+ 9us) server_negotiation.cc:407] Connection header received
0619 09:01:42.866955 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:42.866956 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:42.866969 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:42.867021 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:01:42.867300 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:42.867790 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:42.868545 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:42.868741 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:42.868792 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:42.871038 (+ 2246us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:43691: BlockingRecv error: recv got EOF from 127.0.98.3:43691 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Jun 19 09:01:43 dist-test-slave-mtdr krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 09:01:43 dist-test-slave-mtdr krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 09:01:43.597091 9582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:43.593040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34829 (local address 127.0.98.1:33887)
0619 09:01:43.593157 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0619 09:01:43.593168 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 09:01:43.593238 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:01:43.593378 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:01:43.593379 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:01:43.593521 (+ 142us) client_negotiation.cc:189] Negotiated authn=SASL
0619 09:01:43.593571 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:43.593573 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:43.594168 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:43.594169 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:43.594690 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:43.594691 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:43.594993 (+ 302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:43.594994 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:43.595029 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:43.595416 (+ 387us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 09:01:43.595420 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 09:01:43.597034 (+ 1614us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34829: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":69,"threads_started":1}
W20260619 09:01:43.597190 9583 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 09:01:43.593144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33887 (local address 127.0.98.62:34829)
0619 09:01:43.593257 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:43.593258 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:43.593281 (+ 23us) server_negotiation.cc:407] Connection header received
0619 09:01:43.593300 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:43.593301 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:43.593310 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:43.593367 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:01:43.593620 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:43.594111 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:43.594753 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:43.594948 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:43.594991 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:43.597121 (+ 2130us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:33887: BlockingRecv error: recv got EOF from 127.0.98.1:33887 (error 108)
Metrics: {"server-negotiator.queue_time_us":91,"thread_start_us":52,"threads_started":1}
Jun 19 09:01:43 dist-test-slave-mtdr krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 09:01:43 dist-test-slave-mtdr krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 09:01:43.718039 9584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:43.713533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34829 (local address 127.0.98.2:53561)
0619 09:01:43.713713 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0619 09:01:43.713725 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 09:01:43.713789 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:01:43.713907 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:01:43.713907 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:01:43.714063 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0619 09:01:43.714109 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:43.714111 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:43.714733 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:43.714734 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:43.715359 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:43.715361 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:43.715739 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:43.715740 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:43.715790 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:43.716226 (+ 436us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 09:01:43.716231 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 09:01:43.717978 (+ 1747us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34829: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":69,"threads_started":1}
W20260619 09:01:43.718138 9583 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 09:01:43.713630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53561 (local address 127.0.98.62:34829)
0619 09:01:43.713681 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:43.713683 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:43.713785 (+ 102us) server_negotiation.cc:407] Connection header received
0619 09:01:43.713838 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:43.713839 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:43.713850 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:43.713895 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:01:43.714160 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:43.714664 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:43.715455 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:43.715654 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:43.715720 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:43.718077 (+ 2357us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:53561: BlockingRecv error: recv got EOF from 127.0.98.2:53561 (error 108)
Metrics: {"server-negotiator.queue_time_us":27}
Jun 19 09:01:43 dist-test-slave-mtdr krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 09:01:43 dist-test-slave-mtdr krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 09:01:43.875670 9585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:43.871439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34829 (local address 127.0.98.3:56289)
0619 09:01:43.871589 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0619 09:01:43.871603 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 09:01:43.871666 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:01:43.871786 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:01:43.871787 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:01:43.871979 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0619 09:01:43.872026 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:43.872028 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:43.872628 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:43.872628 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:43.873185 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:43.873187 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:43.873478 (+ 291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:43.873479 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:43.873522 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:43.873936 (+ 414us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 09:01:43.873941 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 09:01:43.875611 (+ 1670us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34829: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":55,"threads_started":1}
W20260619 09:01:43.875751 9583 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 09:01:43.871570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56289 (local address 127.0.98.62:34829)
0619 09:01:43.871635 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:43.871637 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:43.871646 (+ 9us) server_negotiation.cc:407] Connection header received
0619 09:01:43.871716 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:43.871716 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:43.871729 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:43.871774 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:01:43.872085 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:43.872570 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:43.873248 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:43.873422 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:43.873465 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:43.875693 (+ 2228us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:56289: BlockingRecv error: recv got EOF from 127.0.98.3:56289 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Jun 19 09:01:44 dist-test-slave-mtdr krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 09:01:44 dist-test-slave-mtdr krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 09:01:44.601646 9586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:44.597546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34829 (local address 127.0.98.1:47399)
0619 09:01:44.597666 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0619 09:01:44.597678 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 09:01:44.597734 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:01:44.597877 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:01:44.597877 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:01:44.598022 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0619 09:01:44.598067 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:44.598069 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:44.598690 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:44.598691 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:44.599260 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:44.599262 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:44.599591 (+ 329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:44.599591 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:44.599626 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:44.600001 (+ 375us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 09:01:44.600005 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 09:01:44.601588 (+ 1583us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34829: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":53,"threads_started":1}
W20260619 09:01:44.601735 9587 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 09:01:44.597668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47399 (local address 127.0.98.62:34829)
0619 09:01:44.597778 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:44.597780 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:44.597788 (+ 8us) server_negotiation.cc:407] Connection header received
0619 09:01:44.597807 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:44.597808 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:44.597817 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:44.597870 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:01:44.598119 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:44.598607 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:44.599336 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:44.599533 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:44.599587 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:44.601667 (+ 2080us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:47399: BlockingRecv error: recv got EOF from 127.0.98.1:47399 (error 108)
Metrics: {"server-negotiator.queue_time_us":91,"thread_start_us":42,"threads_started":1}
Jun 19 09:01:44 dist-test-slave-mtdr krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 09:01:44 dist-test-slave-mtdr krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 09:01:44.723161 9588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:44.718539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34829 (local address 127.0.98.2:60483)
0619 09:01:44.718651 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0619 09:01:44.718661 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 09:01:44.718723 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:01:44.718862 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:01:44.718862 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:01:44.719029 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0619 09:01:44.719075 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:44.719076 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:44.719896 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:44.719897 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:44.720420 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:44.720422 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:44.720756 (+ 334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:44.720757 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:44.720801 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:44.721312 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 09:01:44.721317 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 09:01:44.723085 (+ 1768us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34829: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":62,"threads_started":1}
W20260619 09:01:44.723225 9587 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 09:01:44.718633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60483 (local address 127.0.98.62:34829)
0619 09:01:44.718679 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:44.718681 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:44.718707 (+ 26us) server_negotiation.cc:407] Connection header received
0619 09:01:44.718773 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:44.718773 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:44.718783 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:44.718839 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:01:44.719154 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:44.719814 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:44.720500 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:44.720689 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:44.720748 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:44.723173 (+ 2425us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:60483: BlockingRecv error: recv got EOF from 127.0.98.2:60483 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
Jun 19 09:01:44 dist-test-slave-mtdr krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 09:01:44 dist-test-slave-mtdr krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 09:01:44.880407 9589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:44.876149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34829 (local address 127.0.98.3:46175)
0619 09:01:44.876268 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0619 09:01:44.876279 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 09:01:44.876338 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:01:44.876442 (+ 104us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:01:44.876443 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:01:44.876592 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0619 09:01:44.876638 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:44.876639 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:44.877229 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:44.877230 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:44.877762 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:44.877764 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:44.878054 (+ 290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:44.878055 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:44.878097 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:44.878527 (+ 430us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 09:01:44.878531 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 09:01:44.880333 (+ 1802us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34829: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":73,"threads_started":1}
W20260619 09:01:44.880498 9587 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 09:01:44.876241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46175 (local address 127.0.98.62:34829)
0619 09:01:44.876285 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:44.876287 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:44.876337 (+ 50us) server_negotiation.cc:407] Connection header received
0619 09:01:44.876386 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:44.876387 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:44.876398 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:44.876433 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:01:44.876687 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:44.877163 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:44.877820 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:44.877999 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:44.878042 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:44.880431 (+ 2389us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:46175: BlockingRecv error: recv got EOF from 127.0.98.3:46175 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
Jun 19 09:01:45 dist-test-slave-mtdr krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 09:01:45 dist-test-slave-mtdr krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 09:01:45.606292 9590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:45.602120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34829 (local address 127.0.98.1:57851)
0619 09:01:45.602240 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0619 09:01:45.602250 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 09:01:45.602306 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:01:45.602442 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:01:45.602443 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:01:45.602594 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0619 09:01:45.602642 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:45.602644 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:45.603298 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:45.603299 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:45.603845 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:45.603847 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:45.604217 (+ 370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:45.604217 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:45.604253 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:45.604652 (+ 399us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 09:01:45.604656 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 09:01:45.606234 (+ 1578us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34829: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":64,"threads_started":1}
W20260619 09:01:45.606393 9591 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 09:01:45.602216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57851 (local address 127.0.98.62:34829)
0619 09:01:45.602315 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:45.602317 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:45.602344 (+ 27us) server_negotiation.cc:407] Connection header received
0619 09:01:45.602366 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:45.602366 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:45.602377 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:45.602430 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:01:45.602692 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:45.603221 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:45.603938 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:45.604149 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:45.604207 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:45.606323 (+ 2116us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:57851: BlockingRecv error: recv got EOF from 127.0.98.1:57851 (error 108)
Metrics: {"server-negotiator.queue_time_us":79,"thread_start_us":42,"threads_started":1}
Jun 19 09:01:45 dist-test-slave-mtdr krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 09:01:45 dist-test-slave-mtdr krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 09:01:45.728430 9592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:45.723670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34829 (local address 127.0.98.2:46125)
0619 09:01:45.723824 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0619 09:01:45.723839 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 09:01:45.723916 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:01:45.724031 (+ 115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:01:45.724032 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:01:45.724198 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0619 09:01:45.724257 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:45.724259 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:45.725014 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:45.725015 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:45.725700 (+ 685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:45.725702 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:45.726040 (+ 338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:45.726041 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:45.726109 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:45.726519 (+ 410us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 09:01:45.726524 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 09:01:45.728347 (+ 1823us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34829: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":66,"threads_started":1}
W20260619 09:01:45.728551 9591 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 09:01:45.723763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46125 (local address 127.0.98.62:34829)
0619 09:01:45.723819 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:45.723820 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:45.723902 (+ 82us) server_negotiation.cc:407] Connection header received
0619 09:01:45.723963 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:45.723964 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:45.723974 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:45.724019 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:01:45.724337 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:45.724862 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:45.725762 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:45.725953 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:45.726025 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:45.728448 (+ 2423us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:46125: BlockingRecv error: recv got EOF from 127.0.98.2:46125 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
Jun 19 09:01:45 dist-test-slave-mtdr krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 09:01:45 dist-test-slave-mtdr krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 09:01:45.888478 9593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:45.880893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34829 (local address 127.0.98.3:57631)
0619 09:01:45.881024 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0619 09:01:45.881036 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 09:01:45.881098 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:01:45.881241 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:01:45.881242 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:01:45.881400 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0619 09:01:45.881444 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:45.881446 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:45.882092 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:45.882093 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:45.882607 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:45.882608 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:45.882975 (+ 367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:45.882976 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:45.883026 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:45.886379 (+ 3353us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 09:01:45.886384 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 09:01:45.888417 (+ 2033us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34829: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":63,"threads_started":1}
W20260619 09:01:45.888549 9591 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 09:01:45.881016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57631 (local address 127.0.98.62:34829)
0619 09:01:45.881092 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:45.881094 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:45.881104 (+ 10us) server_negotiation.cc:407] Connection header received
0619 09:01:45.881161 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:45.881162 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:45.881176 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:45.881225 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:01:45.881512 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:45.882016 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:45.882686 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:45.882909 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:45.882958 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:45.888500 (+ 5542us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:57631: BlockingRecv error: recv got EOF from 127.0.98.3:57631 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Jun 19 09:01:46 dist-test-slave-mtdr krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 09:01:46 dist-test-slave-mtdr krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 09:01:46.611142 9594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:46.606776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34829 (local address 127.0.98.1:37343)
0619 09:01:46.606929 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0619 09:01:46.606942 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 09:01:46.607009 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:01:46.607175 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:01:46.607176 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:01:46.607377 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0619 09:01:46.607428 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:46.607430 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:46.608092 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:46.608094 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:46.608629 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:46.608630 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:46.608946 (+ 316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:46.608947 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:46.608997 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:46.609415 (+ 418us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 09:01:46.609419 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 09:01:46.611077 (+ 1658us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34829: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":80,"threads_started":1}
W20260619 09:01:46.611239 9595 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 09:01:46.606881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37343 (local address 127.0.98.62:34829)
0619 09:01:46.606985 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:46.606987 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:46.606998 (+ 11us) server_negotiation.cc:407] Connection header received
0619 09:01:46.607080 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:46.607081 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:46.607093 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:46.607152 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:01:46.607525 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:46.608013 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:46.608698 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:46.608888 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:46.608942 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:46.611166 (+ 2224us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:37343: BlockingRecv error: recv got EOF from 127.0.98.1:37343 (error 108)
Metrics: {"server-negotiator.queue_time_us":83,"thread_start_us":46,"threads_started":1}
Jun 19 09:01:46 dist-test-slave-mtdr krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 09:01:46 dist-test-slave-mtdr krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 09:01:46.733327 9596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:46.728895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34829 (local address 127.0.98.2:46403)
0619 09:01:46.729071 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0619 09:01:46.729081 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 09:01:46.729182 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:01:46.729323 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:01:46.729324 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:01:46.729489 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0619 09:01:46.729531 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:46.729533 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:46.730160 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:46.730161 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:46.730704 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:46.730706 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:46.731055 (+ 349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:46.731056 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:46.731114 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:46.731540 (+ 426us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 09:01:46.731545 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 09:01:46.733238 (+ 1693us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34829: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":64,"threads_started":1}
W20260619 09:01:46.733393 9595 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 09:01:46.728942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46403 (local address 127.0.98.62:34829)
0619 09:01:46.729021 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:46.729023 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:46.729172 (+ 149us) server_negotiation.cc:407] Connection header received
0619 09:01:46.729246 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:46.729247 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:46.729260 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:46.729309 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:01:46.729594 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:46.730090 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:46.730771 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:46.730996 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:46.731051 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:46.733340 (+ 2289us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:46403: BlockingRecv error: recv got EOF from 127.0.98.2:46403 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Jun 19 09:01:46 dist-test-slave-mtdr krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 09:01:46 dist-test-slave-mtdr krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 09:01:46.893124 9597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:46.888951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34829 (local address 127.0.98.3:46819)
0619 09:01:46.889069 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0619 09:01:46.889078 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 09:01:46.889127 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:01:46.889303 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:01:46.889304 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:01:46.889463 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0619 09:01:46.889506 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:46.889508 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:46.890117 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:46.890118 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:46.890647 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:46.890649 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:46.890971 (+ 322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:46.890972 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:46.891024 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:46.891449 (+ 425us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 09:01:46.891454 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 09:01:46.893065 (+ 1611us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34829: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":53,"threads_started":1}
W20260619 09:01:46.893226 9595 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 09:01:46.889086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46819 (local address 127.0.98.62:34829)
0619 09:01:46.889146 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:46.889148 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:46.889157 (+ 9us) server_negotiation.cc:407] Connection header received
0619 09:01:46.889187 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:46.889187 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:46.889205 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:46.889254 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:01:46.889576 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:46.890062 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:46.890710 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:46.890903 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:46.890949 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:46.893160 (+ 2211us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:46819: BlockingRecv error: recv got EOF from 127.0.98.3:46819 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Jun 19 09:01:47 dist-test-slave-mtdr krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 09:01:47 dist-test-slave-mtdr krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 09:01:47.615948 9598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:47.611601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34829 (local address 127.0.98.1:56275)
0619 09:01:47.611749 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0619 09:01:47.611760 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 09:01:47.611812 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:01:47.611966 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:01:47.611967 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:01:47.612162 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0619 09:01:47.612232 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:47.612234 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:47.612959 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:47.612960 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:47.613491 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:47.613493 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:47.613824 (+ 331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:47.613825 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:47.613860 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:47.614300 (+ 440us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 09:01:47.614306 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0619 09:01:47.615891 (+ 1585us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34829: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":69,"threads_started":1}
W20260619 09:01:47.616037 9599 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 09:01:47.611718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56275 (local address 127.0.98.62:34829)
0619 09:01:47.611819 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:47.611821 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:47.611831 (+ 10us) server_negotiation.cc:407] Connection header received
0619 09:01:47.611874 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:47.611875 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:47.611887 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:47.611948 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:01:47.612304 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:47.612820 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:47.613564 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:47.613773 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:47.613821 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:47.615971 (+ 2150us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:56275: BlockingRecv error: recv got EOF from 127.0.98.1:56275 (error 108)
Metrics: {"server-negotiator.queue_time_us":82,"thread_start_us":57,"threads_started":1}
Jun 19 09:01:47 dist-test-slave-mtdr krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 09:01:47 dist-test-slave-mtdr krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 09:01:47.738359 9600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:47.733855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34829 (local address 127.0.98.2:38955)
0619 09:01:47.733991 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0619 09:01:47.734004 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 09:01:47.734098 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:01:47.734252 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:01:47.734253 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:01:47.734427 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0619 09:01:47.734474 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:47.734477 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:47.735321 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:47.735323 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:47.735860 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:47.735862 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:47.736174 (+ 312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:47.736175 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:47.736227 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:47.736651 (+ 424us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 09:01:47.736656 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 09:01:47.738283 (+ 1627us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34829: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":61,"threads_started":1}
W20260619 09:01:47.738437 9599 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 09:01:47.733982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38955 (local address 127.0.98.62:34829)
0619 09:01:47.734043 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:47.734045 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:47.734071 (+ 26us) server_negotiation.cc:407] Connection header received
0619 09:01:47.734167 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:47.734168 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:47.734181 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:47.734228 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:01:47.734557 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:47.735231 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:47.735931 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:47.736117 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:47.736172 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:47.738385 (+ 2213us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:38955: BlockingRecv error: recv got EOF from 127.0.98.2:38955 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
Jun 19 09:01:47 dist-test-slave-mtdr krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 09:01:47 dist-test-slave-mtdr krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 09:01:47.898115 9601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:47.893629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34829 (local address 127.0.98.3:37089)
0619 09:01:47.893759 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0619 09:01:47.893770 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 09:01:47.893828 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:01:47.893974 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:01:47.893975 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:01:47.894126 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0619 09:01:47.894172 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:47.894174 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:47.894792 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:47.894793 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:47.895357 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:47.895359 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:47.895724 (+ 365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:47.895725 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:47.895786 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:47.896280 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 09:01:47.896285 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 09:01:47.898050 (+ 1765us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34829: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":64,"threads_started":1}
W20260619 09:01:47.898224 9599 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 09:01:47.893699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37089 (local address 127.0.98.62:34829)
0619 09:01:47.893770 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:47.893772 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:47.893815 (+ 43us) server_negotiation.cc:407] Connection header received
0619 09:01:47.893904 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:47.893905 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:47.893917 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:47.893956 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:01:47.894229 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:47.894704 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:47.895448 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:47.895649 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:47.895716 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:47.898149 (+ 2433us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:37089: BlockingRecv error: recv got EOF from 127.0.98.3:37089 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Jun 19 09:01:48 dist-test-slave-mtdr krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 09:01:48 dist-test-slave-mtdr krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 09:01:48.620640 9602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:48.616357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34829 (local address 127.0.98.1:51625)
0619 09:01:48.616464 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0619 09:01:48.616475 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 09:01:48.616526 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:01:48.616668 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:01:48.616669 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:01:48.616818 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0619 09:01:48.616873 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:48.616875 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:48.617505 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:48.617506 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:48.618036 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:48.618037 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:48.618372 (+ 335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:48.618373 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:48.618423 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:48.618872 (+ 449us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 09:01:48.618877 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 09:01:48.620554 (+ 1677us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34829: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":89,"thread_start_us":61,"threads_started":1}
W20260619 09:01:48.620716 9603 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 09:01:48.616442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51625 (local address 127.0.98.62:34829)
0619 09:01:48.616556 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:48.616558 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:48.616566 (+ 8us) server_negotiation.cc:407] Connection header received
0619 09:01:48.616587 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:48.616587 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:48.616597 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:48.616650 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:01:48.616930 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:48.617423 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:48.618093 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:48.618282 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:48.618325 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:48.620654 (+ 2329us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:51625: BlockingRecv error: recv got EOF from 127.0.98.1:51625 (error 108)
Metrics: {"server-negotiator.queue_time_us":94,"thread_start_us":56,"threads_started":1}
Jun 19 09:01:48 dist-test-slave-mtdr krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 09:01:48 dist-test-slave-mtdr krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 09:01:48.743531 9604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:48.738876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34829 (local address 127.0.98.2:52555)
0619 09:01:48.739001 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0619 09:01:48.739013 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 09:01:48.739088 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:01:48.739266 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:01:48.739267 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:01:48.739423 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0619 09:01:48.739475 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:48.739477 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:48.740165 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:48.740166 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:48.740775 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:48.740777 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:48.741122 (+ 345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:48.741123 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:48.741181 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:48.741645 (+ 464us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 09:01:48.741649 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 09:01:48.743427 (+ 1778us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34829: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":65,"threads_started":1}
W20260619 09:01:48.743583 9603 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 09:01:48.738938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52555 (local address 127.0.98.62:34829)
0619 09:01:48.739010 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:48.739011 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:48.739111 (+ 100us) server_negotiation.cc:407] Connection header received
0619 09:01:48.739145 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:48.739146 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:48.739159 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:48.739225 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:01:48.739569 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:48.740077 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:48.740854 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:48.741055 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:48.741106 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:48.743520 (+ 2414us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:52555: BlockingRecv error: recv got EOF from 127.0.98.2:52555 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Jun 19 09:01:48 dist-test-slave-mtdr krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 09:01:48 dist-test-slave-mtdr krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 09:01:48.903012 9605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:48.898583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34829 (local address 127.0.98.3:35091)
0619 09:01:48.898706 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0619 09:01:48.898718 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 09:01:48.898787 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:01:48.898956 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:01:48.898957 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:01:48.899125 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0619 09:01:48.899176 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:48.899178 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:48.899787 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:48.899788 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:48.900322 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:48.900324 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:48.900660 (+ 336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:48.900661 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:48.900706 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:48.901155 (+ 449us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 09:01:48.901160 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 09:01:48.902924 (+ 1764us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34829: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":72,"threads_started":1}
W20260619 09:01:48.903111 9603 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 09:01:48.898681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35091 (local address 127.0.98.62:34829)
0619 09:01:48.898723 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:48.898725 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:48.898778 (+ 53us) server_negotiation.cc:407] Connection header received
0619 09:01:48.898881 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:48.898881 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:48.898893 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:48.898939 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:01:48.899232 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:48.899707 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:48.900405 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:48.900597 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:48.900652 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:48.903045 (+ 2393us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:35091: BlockingRecv error: recv got EOF from 127.0.98.3:35091 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
Jun 19 09:01:49 dist-test-slave-mtdr krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 09:01:49 dist-test-slave-mtdr krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 09:01:49.625427 9606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:49.621114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34829 (local address 127.0.98.1:39793)
0619 09:01:49.621238 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0619 09:01:49.621249 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 09:01:49.621300 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:01:49.621439 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:01:49.621440 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:01:49.621587 (+ 147us) client_negotiation.cc:189] Negotiated authn=SASL
0619 09:01:49.621633 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:49.621634 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:49.622294 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:49.622295 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:49.622863 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:49.622865 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:49.623204 (+ 339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:49.623204 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:49.623244 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:49.623637 (+ 393us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 09:01:49.623641 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 09:01:49.625345 (+ 1704us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34829: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":57,"threads_started":1}
W20260619 09:01:49.625540 9607 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 09:01:49.621216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39793 (local address 127.0.98.62:34829)
0619 09:01:49.621324 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:49.621326 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:49.621333 (+ 7us) server_negotiation.cc:407] Connection header received
0619 09:01:49.621354 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:49.621355 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:49.621366 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:49.621430 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:01:49.621685 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:49.622225 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:49.622942 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:49.623141 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:49.623195 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:49.625453 (+ 2258us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:39793: BlockingRecv error: recv got EOF from 127.0.98.1:39793 (error 108)
Metrics: {"server-negotiator.queue_time_us":91,"thread_start_us":55,"threads_started":1}
Jun 19 09:01:49 dist-test-slave-mtdr krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 09:01:49 dist-test-slave-mtdr krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 09:01:49.748328 9608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:49.744042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34829 (local address 127.0.98.2:42397)
0619 09:01:49.744199 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0619 09:01:49.744210 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 09:01:49.744273 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:01:49.744391 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:01:49.744391 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:01:49.744546 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0619 09:01:49.744589 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:49.744591 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:49.745280 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:49.745281 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:49.745840 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:49.745843 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:49.746186 (+ 343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:49.746187 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:49.746228 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:49.746630 (+ 402us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 09:01:49.746634 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 09:01:49.748243 (+ 1609us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34829: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":66,"threads_started":1}
W20260619 09:01:49.748426 9607 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 09:01:49.744158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42397 (local address 127.0.98.62:34829)
0619 09:01:49.744216 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:49.744218 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:49.744266 (+ 48us) server_negotiation.cc:407] Connection header received
0619 09:01:49.744328 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:49.744328 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:49.744339 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:49.744379 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:01:49.744679 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:49.745195 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:49.745928 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:49.746116 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:49.746182 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:49.748363 (+ 2181us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:42397: BlockingRecv error: recv got EOF from 127.0.98.2:42397 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
Jun 19 09:01:49 dist-test-slave-mtdr krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 09:01:49 dist-test-slave-mtdr krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 09:01:49.908059 9609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:49.903529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34829 (local address 127.0.98.3:49747)
0619 09:01:49.903654 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0619 09:01:49.903664 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 09:01:49.903714 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:01:49.903846 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:01:49.903846 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:01:49.903995 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0619 09:01:49.904040 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:49.904041 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:49.904639 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:49.904640 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:49.905247 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:49.905251 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:49.905664 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:49.905665 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:49.905734 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:49.906218 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 09:01:49.906223 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 09:01:49.907981 (+ 1758us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34829: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":69,"threads_started":1}
W20260619 09:01:49.908171 9607 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 09:01:49.903628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49747 (local address 127.0.98.62:34829)
0619 09:01:49.903681 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:49.903683 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:49.903706 (+ 23us) server_negotiation.cc:407] Connection header received
0619 09:01:49.903763 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:49.903764 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:49.903776 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:49.903827 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:01:49.904096 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:49.904583 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:49.905377 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:49.905565 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:49.905628 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:49.908113 (+ 2485us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:49747: BlockingRecv error: recv got EOF from 127.0.98.3:49747 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
Jun 19 09:01:50 dist-test-slave-mtdr krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 09:01:50 dist-test-slave-mtdr krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 09:01:50.630118 9610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:50.625855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34829 (local address 127.0.98.1:58917)
0619 09:01:50.625969 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0619 09:01:50.625981 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 09:01:50.626050 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:01:50.626180 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:01:50.626180 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:01:50.626329 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0619 09:01:50.626374 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:50.626376 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:50.627044 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:50.627045 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:50.627586 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:50.627588 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:50.627911 (+ 323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:50.627912 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:50.627955 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:50.628383 (+ 428us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 09:01:50.628388 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 09:01:50.630032 (+ 1644us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34829: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":55,"threads_started":1}
W20260619 09:01:50.630199 9611 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 09:01:50.625929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58917 (local address 127.0.98.62:34829)
0619 09:01:50.626060 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:50.626062 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:50.626069 (+ 7us) server_negotiation.cc:407] Connection header received
0619 09:01:50.626089 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:50.626089 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:50.626100 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:50.626174 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:01:50.626426 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:50.626974 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:50.627657 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:50.627849 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:50.627899 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:50.630138 (+ 2239us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:58917: BlockingRecv error: recv got EOF from 127.0.98.1:58917 (error 108)
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":64,"threads_started":1}
Jun 19 09:01:50 dist-test-slave-mtdr krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 09:01:50 dist-test-slave-mtdr krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 09:01:50.752889 9612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:50.748806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34829 (local address 127.0.98.2:41795)
0619 09:01:50.748926 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0619 09:01:50.748936 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 09:01:50.748987 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:01:50.749100 (+ 113us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:01:50.749101 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:01:50.749263 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0619 09:01:50.749310 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:50.749312 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:50.749900 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:50.749900 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:50.750418 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:50.750420 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:50.750693 (+ 273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:50.750694 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:50.750731 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:50.751175 (+ 444us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 09:01:50.751181 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0619 09:01:50.752830 (+ 1649us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34829: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":56,"threads_started":1}
W20260619 09:01:50.752978 9611 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 09:01:50.748866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41795 (local address 127.0.98.62:34829)
0619 09:01:50.748919 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:50.748921 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:50.748974 (+ 53us) server_negotiation.cc:407] Connection header received
0619 09:01:50.749030 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:50.749031 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:50.749041 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:50.749089 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:01:50.749357 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:50.749847 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:50.750469 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:50.750647 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:50.750689 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:50.752920 (+ 2231us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:41795: BlockingRecv error: recv got EOF from 127.0.98.2:41795 (error 108)
Metrics: {"server-negotiator.queue_time_us":29}
Jun 19 09:01:50 dist-test-slave-mtdr krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 09:01:50 dist-test-slave-mtdr krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 09:01:50.916841 9613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:50.908486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34829 (local address 127.0.98.3:60737)
0619 09:01:50.908631 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0619 09:01:50.908645 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 09:01:50.908715 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:01:50.908839 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:01:50.908840 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:01:50.908997 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0619 09:01:50.909048 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:50.909050 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:50.909685 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:50.909686 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:50.910224 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:50.910226 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:50.910600 (+ 374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:50.910601 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:50.910652 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:50.915042 (+ 4390us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 09:01:50.915048 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0619 09:01:50.916780 (+ 1732us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34829: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":54,"threads_started":1}
W20260619 09:01:50.916946 9611 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 09:01:50.908587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60737 (local address 127.0.98.62:34829)
0619 09:01:50.908666 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:50.908668 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:50.908696 (+ 28us) server_negotiation.cc:407] Connection header received
0619 09:01:50.908766 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:50.908767 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:50.908778 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:50.908823 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:01:50.909110 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:50.909609 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:50.910302 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:50.910487 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:50.910538 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:50.916876 (+ 6338us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:60737: BlockingRecv error: recv got EOF from 127.0.98.3:60737 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Jun 19 09:01:51 dist-test-slave-mtdr krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 09:01:51 dist-test-slave-mtdr krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 09:01:51.635008 9614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:51.630627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34829 (local address 127.0.98.1:45923)
0619 09:01:51.630742 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0619 09:01:51.630753 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 09:01:51.630800 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:01:51.630942 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:01:51.630942 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:01:51.631115 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0619 09:01:51.631164 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:51.631166 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:51.631811 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:51.631813 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:51.632381 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:51.632383 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:51.632753 (+ 370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:51.632754 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:51.632805 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:51.633264 (+ 459us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 09:01:51.633269 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 09:01:51.634930 (+ 1661us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34829: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":64,"threads_started":1}
W20260619 09:01:51.635082 9615 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 09:01:51.630700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45923 (local address 127.0.98.62:34829)
0619 09:01:51.630808 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:51.630809 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:51.630854 (+ 45us) server_negotiation.cc:407] Connection header received
0619 09:01:51.630875 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:51.630875 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:51.630884 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:51.630932 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:01:51.631217 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:51.631714 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:51.632466 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:51.632667 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:51.632721 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:51.635019 (+ 2298us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:45923: BlockingRecv error: recv got EOF from 127.0.98.1:45923 (error 108)
Metrics: {"server-negotiator.queue_time_us":88,"thread_start_us":52,"threads_started":1}
Jun 19 09:01:51 dist-test-slave-mtdr krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 09:01:51 dist-test-slave-mtdr krb5kdc[8556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 09:01:51.758030 9616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:51.753456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34829 (local address 127.0.98.2:58661)
0619 09:01:51.753604 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0619 09:01:51.753615 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 09:01:51.753680 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:01:51.753798 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:01:51.753799 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:01:51.753950 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0619 09:01:51.753996 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:51.753998 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:51.754695 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:51.754696 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:51.755497 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:51.755500 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:51.755827 (+ 327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:51.755828 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:51.755885 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:51.756304 (+ 419us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 09:01:51.756308 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 09:01:51.757931 (+ 1623us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34829: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":68,"threads_started":1}
W20260619 09:01:51.758083 9615 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 09:01:51.753560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58661 (local address 127.0.98.62:34829)
0619 09:01:51.753609 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:51.753612 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:51.753679 (+ 67us) server_negotiation.cc:407] Connection header received
0619 09:01:51.753728 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:51.753728 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:51.753739 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:51.753785 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:01:51.754106 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:51.754612 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:51.755558 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:51.755767 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:51.755820 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:51.758023 (+ 2203us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:58661: BlockingRecv error: recv got EOF from 127.0.98.2:58661 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
I20260619 09:01:51.830179 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ez5NB/build/release/bin/kudu with pid 9139
I20260619 09:01:51.834050 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ez5NB/build/release/bin/kudu with pid 9290
I20260619 09:01:51.837885 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ez5NB/build/release/bin/kudu with pid 9425
I20260619 09:01:51.841292 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ez5NB/build/release/bin/kudu with pid 9069
2026-06-19T09:01:51Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17275 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-06-19T09:01:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T09:01:51Z Disabled control of system clock
I20260619 09:01:51.865722 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ez5NB/build/release/bin/kudu
/tmp/dist-test-task4ez5NB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781859658134510-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781859658134510-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781859658134510-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781859658134510-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38683
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:36409
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38683
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260619 09:01:51.958583 9620 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 09:01:51.958763 9620 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 09:01:51.958781 9620 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 09:01:51.959913 9620 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 09:01:51.959952 9620 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 09:01:51.959964 9620 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 09:01:51.959977 9620 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 09:01:51.960000 9620 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 09:01:51.961326 9620 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36409
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781859658134510-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781859658134510-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38683
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38683
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781859658134510-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9620
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781859658134510-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 09:01:51.961539 9620 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 09:01:51.961798 9620 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 09:01:51.966421 9628 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:01:51.966377 9626 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:01:51.966434 9620 server_base.cc:1034] running on GCE node
W20260619 09:01:51.966401 9625 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:01:51.966954 9620 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 09:01:51.967181 9620 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 09:01:51.968336 9620 hybrid_clock.cc:648] HybridClock initialized: now 1781859711968322 us; error 26 us; skew 500 ppm
I20260619 09:01:51.969498 9620 webserver.cc:460] Webserver started at http://127.0.98.62:38795/ using document root <none> and password file <none>
I20260619 09:01:51.969703 9620 fs_manager.cc:359] Metadata directory not provided
I20260619 09:01:51.969744 9620 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 09:01:51.969877 9620 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 09:01:51.970862 9620 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781859658134510-392-0/minicluster-data/master-0/data/instance:
uuid: "94b169a35a524d7d88787adfaf6864e6"
format_stamp: "Formatted at 2026-06-19 09:01:51 on dist-test-slave-mtdr"
server_key: "3b90ff92622cfde87c4a440e4a664a65"
server_key_iv: "84dc267a2540c3c15e855569881aab83"
server_key_version: "encryptionkey@0"
I20260619 09:01:51.971194 9620 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781859658134510-392-0/minicluster-data/master-0/wal/instance:
uuid: "94b169a35a524d7d88787adfaf6864e6"
format_stamp: "Formatted at 2026-06-19 09:01:51 on dist-test-slave-mtdr"
server_key: "3b90ff92622cfde87c4a440e4a664a65"
server_key_iv: "84dc267a2540c3c15e855569881aab83"
server_key_version: "encryptionkey@0"
I20260619 09:01:51.972401 9620 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260619 09:01:51.973060 9634 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 09:01:51.973217 9620 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 09:01:51.973299 9620 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781859658134510-392-0/minicluster-data/master-0/data,/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781859658134510-392-0/minicluster-data/master-0/wal
uuid: "94b169a35a524d7d88787adfaf6864e6"
format_stamp: "Formatted at 2026-06-19 09:01:51 on dist-test-slave-mtdr"
server_key: "3b90ff92622cfde87c4a440e4a664a65"
server_key_iv: "84dc267a2540c3c15e855569881aab83"
server_key_version: "encryptionkey@0"
I20260619 09:01:51.973351 9620 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781859658134510-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781859658134510-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781859658134510-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 09:01:51.994243 9620 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 09:01:51.994484 9620 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 09:01:51.994577 9620 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 09:01:51.997934 9620 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38683
I20260619 09:01:51.997987 9686 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38683 every 8 connection(s)
I20260619 09:01:51.998258 9620 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781859658134510-392-0/minicluster-data/master-0/data/info.pb
I20260619 09:01:51.999012 9687 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 09:01:51.999693 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ez5NB/build/release/bin/kudu as pid 9620
I20260619 09:01:51.999796 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781859658134510-392-0/minicluster-data/master-0/wal/instance
I20260619 09:01:52.000008 392 external_mini_cluster.cc:1428] Setting key 11bad5b84806d7c256606e24604c604f
I20260619 09:01:52.002115 9687 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 94b169a35a524d7d88787adfaf6864e6: Bootstrap starting.
I20260619 09:01:52.002774 9687 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 94b169a35a524d7d88787adfaf6864e6: Neither blocks nor log segments found. Creating new log.
I20260619 09:01:52.003098 9687 log.cc:824] T 00000000000000000000000000000000 P 94b169a35a524d7d88787adfaf6864e6: Log is configured to *not* fsync() on all Append() calls
I20260619 09:01:52.003818 9687 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 94b169a35a524d7d88787adfaf6864e6: No bootstrap required, opened a new log
I20260619 09:01:52.005208 9690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:52.000513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34662 (local address 127.0.98.62:38683)
0619 09:01:52.000682 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:52.000689 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:52.000711 (+ 22us) server_negotiation.cc:407] Connection header received
0619 09:01:52.001114 (+ 403us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:52.001120 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:52.001177 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:52.001283 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:01:52.001513 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:52.002378 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:52.003450 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:52.003920 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:52.004032 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:52.004654 (+ 622us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:01:52.004658 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:01:52.004658 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 09:01:52.004817 (+ 159us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:01:52.004846 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 09:01:52.004857 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:01:52.004908 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0619 09:01:52.004982 (+ 74us) server_negotiation.cc:299] Negotiation successful
0619 09:01:52.005054 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":6,"server-negotiator.queue_time_us":106,"thread_start_us":58,"threads_started":1}
I20260619 09:01:52.005769 9687 raft_consensus.cc:348] T 00000000000000000000000000000000 P 94b169a35a524d7d88787adfaf6864e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94b169a35a524d7d88787adfaf6864e6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38683 } }
I20260619 09:01:52.005925 9687 raft_consensus.cc:374] T 00000000000000000000000000000000 P 94b169a35a524d7d88787adfaf6864e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 09:01:52.005961 9687 raft_consensus.cc:729] T 00000000000000000000000000000000 P 94b169a35a524d7d88787adfaf6864e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94b169a35a524d7d88787adfaf6864e6, State: Initialized, Role: FOLLOWER
I20260619 09:01:52.006052 9687 consensus_queue.cc:260] T 00000000000000000000000000000000 P 94b169a35a524d7d88787adfaf6864e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94b169a35a524d7d88787adfaf6864e6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38683 } }
I20260619 09:01:52.006105 9687 raft_consensus.cc:388] T 00000000000000000000000000000000 P 94b169a35a524d7d88787adfaf6864e6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 09:01:52.006127 9687 raft_consensus.cc:482] T 00000000000000000000000000000000 P 94b169a35a524d7d88787adfaf6864e6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 09:01:52.006160 9687 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 94b169a35a524d7d88787adfaf6864e6 [term 0 FOLLOWER]: Advancing to term 1
I20260619 09:01:52.006645 9687 raft_consensus.cc:504] T 00000000000000000000000000000000 P 94b169a35a524d7d88787adfaf6864e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94b169a35a524d7d88787adfaf6864e6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38683 } }
I20260619 09:01:52.006731 9687 leader_election.cc:302] T 00000000000000000000000000000000 P 94b169a35a524d7d88787adfaf6864e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 94b169a35a524d7d88787adfaf6864e6; no voters:
I20260619 09:01:52.006880 9687 leader_election.cc:288] T 00000000000000000000000000000000 P 94b169a35a524d7d88787adfaf6864e6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 09:01:52.006920 9692 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 94b169a35a524d7d88787adfaf6864e6 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 09:01:52.007149 9687 sys_catalog.cc:564] T 00000000000000000000000000000000 P 94b169a35a524d7d88787adfaf6864e6 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 09:01:52.007220 9692 raft_consensus.cc:686] T 00000000000000000000000000000000 P 94b169a35a524d7d88787adfaf6864e6 [term 1 LEADER]: Becoming Leader. State: Replica: 94b169a35a524d7d88787adfaf6864e6, State: Running, Role: LEADER
I20260619 09:01:52.007392 9692 consensus_queue.cc:237] T 00000000000000000000000000000000 P 94b169a35a524d7d88787adfaf6864e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94b169a35a524d7d88787adfaf6864e6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38683 } }
I20260619 09:01:52.007826 9694 sys_catalog.cc:455] T 00000000000000000000000000000000 P 94b169a35a524d7d88787adfaf6864e6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 94b169a35a524d7d88787adfaf6864e6. Latest consensus state: current_term: 1 leader_uuid: "94b169a35a524d7d88787adfaf6864e6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94b169a35a524d7d88787adfaf6864e6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38683 } } }
I20260619 09:01:52.007858 9693 sys_catalog.cc:455] T 00000000000000000000000000000000 P 94b169a35a524d7d88787adfaf6864e6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "94b169a35a524d7d88787adfaf6864e6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94b169a35a524d7d88787adfaf6864e6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38683 } } }
I20260619 09:01:52.007918 9694 sys_catalog.cc:458] T 00000000000000000000000000000000 P 94b169a35a524d7d88787adfaf6864e6 [sys.catalog]: This master's current role is: LEADER
I20260619 09:01:52.007923 9693 sys_catalog.cc:458] T 00000000000000000000000000000000 P 94b169a35a524d7d88787adfaf6864e6 [sys.catalog]: This master's current role is: LEADER
I20260619 09:01:52.008100 9700 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 09:01:52.008558 9700 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 09:01:52.009974 9700 catalog_manager.cc:1347] Generated new cluster ID: 0b2c797e33e144e39521d53b1f4111db
I20260619 09:01:52.010025 9700 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 09:01:52.019924 9700 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 09:01:52.020433 9700 catalog_manager.cc:1504] Loading token signing keys...
I20260619 09:01:52.024027 9700 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 94b169a35a524d7d88787adfaf6864e6: Generated new TSK 0
I20260619 09:01:52.024176 9700 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 09:01:52.026346 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ez5NB/build/release/bin/kudu
/tmp/dist-test-task4ez5NB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781859658134510-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781859658134510-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781859658134510-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781859658134510-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38683
--builtin_ntp_servers=127.0.98.20:36409
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260619 09:01:52.121971 9711 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 09:01:52.122159 9711 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 09:01:52.122186 9711 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 09:01:52.123328 9711 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 09:01:52.123366 9711 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 09:01:52.123397 9711 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 09:01:52.124671 9711 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36409
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781859658134510-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781859658134510-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781859658134510-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38683
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9711
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781859658134510-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 09:01:52.124883 9711 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 09:01:52.125128 9711 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 09:01:52.130613 9716 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:01:52.130666 9717 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:01:52.130620 9719 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:01:52.130903 9711 server_base.cc:1034] running on GCE node
I20260619 09:01:52.131080 9711 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 09:01:52.131278 9711 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 09:01:52.132426 9711 hybrid_clock.cc:648] HybridClock initialized: now 1781859712132412 us; error 28 us; skew 500 ppm
I20260619 09:01:52.133570 9711 webserver.cc:460] Webserver started at http://127.0.98.1:44373/ using document root <none> and password file <none>
I20260619 09:01:52.133769 9711 fs_manager.cc:359] Metadata directory not provided
I20260619 09:01:52.133816 9711 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 09:01:52.133940 9711 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 09:01:52.134964 9711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781859658134510-392-0/minicluster-data/ts-0/data/instance:
uuid: "c2390f93d1ac4554b606bc48752467ce"
format_stamp: "Formatted at 2026-06-19 09:01:52 on dist-test-slave-mtdr"
server_key: "7c158179eca13d15ba2cd420aea51980"
server_key_iv: "7ba4fd8ddd9b4c2cb0a5b925a6ba3f75"
server_key_version: "encryptionkey@0"
I20260619 09:01:52.135268 9711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781859658134510-392-0/minicluster-data/ts-0/wal/instance:
uuid: "c2390f93d1ac4554b606bc48752467ce"
format_stamp: "Formatted at 2026-06-19 09:01:52 on dist-test-slave-mtdr"
server_key: "7c158179eca13d15ba2cd420aea51980"
server_key_iv: "7ba4fd8ddd9b4c2cb0a5b925a6ba3f75"
server_key_version: "encryptionkey@0"
I20260619 09:01:52.136458 9711 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260619 09:01:52.137137 9725 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 09:01:52.137281 9711 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 09:01:52.137355 9711 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781859658134510-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781859658134510-392-0/minicluster-data/ts-0/wal
uuid: "c2390f93d1ac4554b606bc48752467ce"
format_stamp: "Formatted at 2026-06-19 09:01:52 on dist-test-slave-mtdr"
server_key: "7c158179eca13d15ba2cd420aea51980"
server_key_iv: "7ba4fd8ddd9b4c2cb0a5b925a6ba3f75"
server_key_version: "encryptionkey@0"
I20260619 09:01:52.137436 9711 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781859658134510-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781859658134510-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781859658134510-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 09:01:52.145998 9711 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 09:01:52.146227 9711 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 09:01:52.146340 9711 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 09:01:52.146531 9711 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 09:01:52.146799 9711 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 09:01:52.146854 9711 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 09:01:52.146891 9711 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 09:01:52.146916 9711 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 09:01:52.151975 9711 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37569
I20260619 09:01:52.152029 9838 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37569 every 8 connection(s)
I20260619 09:01:52.152297 9711 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781859658134510-392-0/minicluster-data/ts-0/data/info.pb
I20260619 09:01:52.157126 9690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:52.152962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58345 (local address 127.0.98.62:38683)
0619 09:01:52.153025 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:52.153028 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:52.153230 (+ 202us) server_negotiation.cc:407] Connection header received
0619 09:01:52.153797 (+ 567us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:52.153798 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:52.153810 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:52.153859 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:01:52.154559 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:52.155103 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:52.156074 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:52.156252 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:52.156292 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:52.156943 (+ 651us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:01:52.156944 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:01:52.156944 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 09:01:52.156948 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:01:52.156962 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 09:01:52.156965 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:01:52.157001 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0619 09:01:52.157051 (+ 50us) server_negotiation.cc:299] Negotiation successful
0619 09:01:52.157072 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260619 09:01:52.157289 9841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:52.152941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38683 (local address 127.0.98.1:58345)
0619 09:01:52.153106 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0619 09:01:52.153145 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 09:01:52.153707 (+ 562us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:01:52.153923 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:01:52.153927 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:01:52.153936 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0619 09:01:52.154485 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:52.154491 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:52.155183 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:52.155184 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:52.156013 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:52.156015 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:52.156303 (+ 288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:52.156304 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:52.156401 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:52.156833 (+ 432us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 09:01:52.156834 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 09:01:52.156859 (+ 25us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 09:01:52.156880 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 09:01:52.157001 (+ 121us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 09:01:52.157003 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 09:01:52.157006 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 09:01:52.157047 (+ 41us) client_negotiation.cc:240] Negotiation successful
0619 09:01:52.157116 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":75,"threads_started":1}
I20260619 09:01:52.157717 9839 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38683
I20260619 09:01:52.157793 9839 heartbeater.cc:464] Registering TS with master...
I20260619 09:01:52.157924 9839 heartbeater.cc:511] Master 127.0.98.62:38683 requested a full tablet report, sending...
I20260619 09:01:52.158362 9651 ts_manager.cc:194] Registered new tserver with Master: c2390f93d1ac4554b606bc48752467ce (127.0.98.1:37569)
I20260619 09:01:52.158996 9651 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:58345
I20260619 09:01:52.161291 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ez5NB/build/release/bin/kudu as pid 9711
I20260619 09:01:52.161369 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781859658134510-392-0/minicluster-data/ts-0/wal/instance
I20260619 09:01:52.161537 392 external_mini_cluster.cc:1428] Setting key 563fab53c68b173f9006fe0a848f33aa
I20260619 09:01:52.162295 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ez5NB/build/release/bin/kudu
/tmp/dist-test-task4ez5NB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781859658134510-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781859658134510-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781859658134510-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781859658134510-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38683
--builtin_ntp_servers=127.0.98.20:36409
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260619 09:01:52.242030 9842 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 09:01:52.242215 9842 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 09:01:52.242241 9842 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 09:01:52.243356 9842 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 09:01:52.243392 9842 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 09:01:52.243424 9842 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 09:01:52.244683 9842 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36409
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781859658134510-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781859658134510-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781859658134510-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38683
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9842
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781859658134510-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 09:01:52.244895 9842 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 09:01:52.245149 9842 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 09:01:52.249591 9850 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:01:52.249609 9848 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:01:52.249643 9847 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:01:52.249913 9842 server_base.cc:1034] running on GCE node
I20260619 09:01:52.250111 9842 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 09:01:52.250312 9842 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 09:01:52.251438 9842 hybrid_clock.cc:648] HybridClock initialized: now 1781859712251419 us; error 30 us; skew 500 ppm
I20260619 09:01:52.252509 9842 webserver.cc:460] Webserver started at http://127.0.98.2:44111/ using document root <none> and password file <none>
I20260619 09:01:52.252723 9842 fs_manager.cc:359] Metadata directory not provided
I20260619 09:01:52.252771 9842 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 09:01:52.252889 9842 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 09:01:52.253865 9842 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781859658134510-392-0/minicluster-data/ts-1/data/instance:
uuid: "48daa4fb94b7426d95ed8a1d5c1a1ddf"
format_stamp: "Formatted at 2026-06-19 09:01:52 on dist-test-slave-mtdr"
server_key: "713e3315574fb548d21a07b4ef30aaa7"
server_key_iv: "a3145af66af933f7358bbf74baa212ff"
server_key_version: "encryptionkey@0"
I20260619 09:01:52.254171 9842 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781859658134510-392-0/minicluster-data/ts-1/wal/instance:
uuid: "48daa4fb94b7426d95ed8a1d5c1a1ddf"
format_stamp: "Formatted at 2026-06-19 09:01:52 on dist-test-slave-mtdr"
server_key: "713e3315574fb548d21a07b4ef30aaa7"
server_key_iv: "a3145af66af933f7358bbf74baa212ff"
server_key_version: "encryptionkey@0"
I20260619 09:01:52.255370 9842 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 09:01:52.256083 9856 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 09:01:52.256224 9842 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 09:01:52.256291 9842 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781859658134510-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781859658134510-392-0/minicluster-data/ts-1/wal
uuid: "48daa4fb94b7426d95ed8a1d5c1a1ddf"
format_stamp: "Formatted at 2026-06-19 09:01:52 on dist-test-slave-mtdr"
server_key: "713e3315574fb548d21a07b4ef30aaa7"
server_key_iv: "a3145af66af933f7358bbf74baa212ff"
server_key_version: "encryptionkey@0"
I20260619 09:01:52.256361 9842 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781859658134510-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781859658134510-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781859658134510-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 09:01:52.265661 9842 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 09:01:52.265882 9842 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 09:01:52.265987 9842 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 09:01:52.266167 9842 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 09:01:52.266434 9842 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 09:01:52.266464 9842 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 09:01:52.266489 9842 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 09:01:52.266511 9842 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 09:01:52.271695 9842 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43701
I20260619 09:01:52.271731 9969 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43701 every 8 connection(s)
I20260619 09:01:52.271987 9842 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781859658134510-392-0/minicluster-data/ts-1/data/info.pb
I20260619 09:01:52.276320 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ez5NB/build/release/bin/kudu as pid 9842
I20260619 09:01:52.276436 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781859658134510-392-0/minicluster-data/ts-1/wal/instance
I20260619 09:01:52.276641 392 external_mini_cluster.cc:1428] Setting key 5b14193f7d659f62f8302d9ec51a808d
I20260619 09:01:52.276857 9690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:52.272602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56361 (local address 127.0.98.62:38683)
0619 09:01:52.272654 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:52.272656 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:52.272867 (+ 211us) server_negotiation.cc:407] Connection header received
0619 09:01:52.273308 (+ 441us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:52.273309 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:52.273318 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:52.273373 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:01:52.273982 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:52.274445 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:52.275449 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:52.275651 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:52.275710 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:52.276574 (+ 864us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:01:52.276575 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:01:52.276575 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 09:01:52.276580 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:01:52.276595 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 09:01:52.276599 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:01:52.276647 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0619 09:01:52.276754 (+ 107us) server_negotiation.cc:299] Negotiation successful
0619 09:01:52.276776 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260619 09:01:52.277088 9972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:52.272576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38683 (local address 127.0.98.2:56361)
0619 09:01:52.272754 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0619 09:01:52.272791 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0619 09:01:52.273226 (+ 435us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:01:52.273430 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:01:52.273434 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:01:52.273449 (+ 15us) client_negotiation.cc:189] Negotiated authn=SASL
0619 09:01:52.273920 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:52.273925 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:52.274508 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:52.274509 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:52.275382 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:52.275385 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:52.275753 (+ 368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:52.275754 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:52.275887 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:52.276420 (+ 533us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 09:01:52.276421 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 09:01:52.276457 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 09:01:52.276481 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 09:01:52.276661 (+ 180us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 09:01:52.276667 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 09:01:52.276671 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 09:01:52.276750 (+ 79us) client_negotiation.cc:240] Negotiation successful
0619 09:01:52.276873 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":70,"threads_started":1}
I20260619 09:01:52.277472 9970 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38683
I20260619 09:01:52.277484 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ez5NB/build/release/bin/kudu
/tmp/dist-test-task4ez5NB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781859658134510-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781859658134510-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781859658134510-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781859658134510-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38683
--builtin_ntp_servers=127.0.98.20:36409
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260619 09:01:52.277555 9970 heartbeater.cc:464] Registering TS with master...
I20260619 09:01:52.277731 9970 heartbeater.cc:511] Master 127.0.98.62:38683 requested a full tablet report, sending...
I20260619 09:01:52.278061 9651 ts_manager.cc:194] Registered new tserver with Master: 48daa4fb94b7426d95ed8a1d5c1a1ddf (127.0.98.2:43701)
I20260619 09:01:52.278411 9651 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:56361
W20260619 09:01:52.354133 9973 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 09:01:52.354313 9973 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 09:01:52.354331 9973 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 09:01:52.355430 9973 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 09:01:52.355474 9973 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 09:01:52.355496 9973 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 09:01:52.356751 9973 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36409
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781859658134510-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781859658134510-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781859658134510-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38683
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9973
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781859658134510-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 09:01:52.356961 9973 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 09:01:52.357223 9973 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 09:01:52.361785 9981 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:01:52.361799 9978 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:01:52.362015 9973 server_base.cc:1034] running on GCE node
W20260619 09:01:52.361919 9979 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:01:52.362215 9973 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 09:01:52.362411 9973 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 09:01:52.363571 9973 hybrid_clock.cc:648] HybridClock initialized: now 1781859712363541 us; error 25 us; skew 500 ppm
I20260619 09:01:52.364684 9973 webserver.cc:460] Webserver started at http://127.0.98.3:46207/ using document root <none> and password file <none>
I20260619 09:01:52.364874 9973 fs_manager.cc:359] Metadata directory not provided
I20260619 09:01:52.364915 9973 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 09:01:52.365024 9973 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 09:01:52.365991 9973 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781859658134510-392-0/minicluster-data/ts-2/data/instance:
uuid: "ea90fb1e91a844ff9db6c88430e24d05"
format_stamp: "Formatted at 2026-06-19 09:01:52 on dist-test-slave-mtdr"
server_key: "ece35f6a72d2dad4aa779b3b235befc5"
server_key_iv: "14525a8712461a08520076e409f615a9"
server_key_version: "encryptionkey@0"
I20260619 09:01:52.366289 9973 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781859658134510-392-0/minicluster-data/ts-2/wal/instance:
uuid: "ea90fb1e91a844ff9db6c88430e24d05"
format_stamp: "Formatted at 2026-06-19 09:01:52 on dist-test-slave-mtdr"
server_key: "ece35f6a72d2dad4aa779b3b235befc5"
server_key_iv: "14525a8712461a08520076e409f615a9"
server_key_version: "encryptionkey@0"
I20260619 09:01:52.367466 9973 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 09:01:52.368088 9987 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 09:01:52.368218 9973 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 09:01:52.368295 9973 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781859658134510-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781859658134510-392-0/minicluster-data/ts-2/wal
uuid: "ea90fb1e91a844ff9db6c88430e24d05"
format_stamp: "Formatted at 2026-06-19 09:01:52 on dist-test-slave-mtdr"
server_key: "ece35f6a72d2dad4aa779b3b235befc5"
server_key_iv: "14525a8712461a08520076e409f615a9"
server_key_version: "encryptionkey@0"
I20260619 09:01:52.368351 9973 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781859658134510-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781859658134510-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781859658134510-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 09:01:52.385097 9973 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 09:01:52.385344 9973 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 09:01:52.385464 9973 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 09:01:52.385665 9973 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 09:01:52.385941 9973 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 09:01:52.385973 9973 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 09:01:52.385999 9973 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 09:01:52.386023 9973 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 09:01:52.391145 9973 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45183
I20260619 09:01:52.391201 10100 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45183 every 8 connection(s)
I20260619 09:01:52.391465 9973 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781859658134510-392-0/minicluster-data/ts-2/data/info.pb
I20260619 09:01:52.391935 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ez5NB/build/release/bin/kudu as pid 9973
I20260619 09:01:52.392028 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781859658134510-392-0/minicluster-data/ts-2/wal/instance
I20260619 09:01:52.392225 392 external_mini_cluster.cc:1428] Setting key c6c9754058f8f0fe805db1110971c5ef
I20260619 09:01:52.396193 9690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:52.392129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50823 (local address 127.0.98.62:38683)
0619 09:01:52.392170 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:52.392172 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:52.392466 (+ 294us) server_negotiation.cc:407] Connection header received
0619 09:01:52.392985 (+ 519us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:52.392986 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:52.393006 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:52.393057 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:01:52.393677 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:52.394170 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:52.395132 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:52.395315 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:52.395361 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:52.395983 (+ 622us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:01:52.395984 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:01:52.395985 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 09:01:52.395989 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:01:52.396011 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 09:01:52.396014 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:01:52.396053 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0619 09:01:52.396120 (+ 67us) server_negotiation.cc:299] Negotiation successful
0619 09:01:52.396138 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260619 09:01:52.396380 10103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:52.392137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38683 (local address 127.0.98.3:50823)
0619 09:01:52.392334 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0619 09:01:52.392375 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 09:01:52.392916 (+ 541us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:01:52.393133 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:01:52.393137 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:01:52.393146 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0619 09:01:52.393597 (+ 451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:52.393602 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:52.394235 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:52.394235 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:52.395052 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:52.395054 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:52.395366 (+ 312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:52.395367 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:52.395462 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:52.395873 (+ 411us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 09:01:52.395874 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 09:01:52.395899 (+ 25us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 09:01:52.395919 (+ 20us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 09:01:52.396058 (+ 139us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 09:01:52.396061 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 09:01:52.396063 (+ 2us) client_negotiation.cc:769] Sending connection context
0619 09:01:52.396115 (+ 52us) client_negotiation.cc:240] Negotiation successful
0619 09:01:52.396186 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":88,"threads_started":1}
I20260619 09:01:52.396775 10101 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38683
I20260619 09:01:52.396849 10101 heartbeater.cc:464] Registering TS with master...
I20260619 09:01:52.397006 10101 heartbeater.cc:511] Master 127.0.98.62:38683 requested a full tablet report, sending...
I20260619 09:01:52.397310 9651 ts_manager.cc:194] Registered new tserver with Master: ea90fb1e91a844ff9db6c88430e24d05 (127.0.98.3:45183)
I20260619 09:01:52.397644 9651 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:50823
I20260619 09:01:52.403738 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260619 09:01:52.405408 9690 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 09:01:52.404961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34666 (local address 127.0.98.62:38683)
0619 09:01:52.405022 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:52.405023 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:52.405115 (+ 92us) server_negotiation.cc:407] Connection header received
0619 09:01:52.405175 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:52.405176 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:52.405184 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:52.405225 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:01:52.405355 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34666: BlockingRecv error: recv got EOF from 127.0.0.1:34666 (error 108)
Metrics: {"server-negotiator.queue_time_us":28}
I20260619 09:01:52.405445 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ez5NB/build/release/bin/kudu with pid 9711
I20260619 09:01:52.409583 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ez5NB/build/release/bin/kudu with pid 9842
I20260619 09:01:52.413216 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ez5NB/build/release/bin/kudu with pid 9973
I20260619 09:01:52.416864 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ez5NB/build/release/bin/kudu with pid 9620
2026-06-19T09:01:52Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (574 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-06-19T09:01:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T09:01:52Z Disabled control of system clock
I20260619 09:01:52.440845 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ez5NB/build/release/bin/kudu
/tmp/dist-test-task4ez5NB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781859658134510-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781859658134510-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781859658134510-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781859658134510-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34151
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:36495
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34151
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260619 09:01:52.516933 10112 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 09:01:52.517125 10112 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 09:01:52.517153 10112 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 09:01:52.518231 10112 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 09:01:52.518278 10112 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 09:01:52.518296 10112 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 09:01:52.518311 10112 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 09:01:52.518337 10112 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 09:01:52.519690 10112 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36495
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781859658134510-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781859658134510-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34151
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34151
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781859658134510-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10112
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781859658134510-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 09:01:52.519912 10112 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 09:01:52.520155 10112 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 09:01:52.524410 10117 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:01:52.524408 10120 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:01:52.524498 10118 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:01:52.524760 10112 server_base.cc:1034] running on GCE node
I20260619 09:01:52.524986 10112 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 09:01:52.525226 10112 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 09:01:52.526368 10112 hybrid_clock.cc:648] HybridClock initialized: now 1781859712526352 us; error 26 us; skew 500 ppm
I20260619 09:01:52.527495 10112 webserver.cc:460] Webserver started at http://127.0.98.62:35733/ using document root <none> and password file <none>
I20260619 09:01:52.527719 10112 fs_manager.cc:359] Metadata directory not provided
I20260619 09:01:52.527765 10112 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 09:01:52.527882 10112 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 09:01:52.528842 10112 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781859658134510-392-0/minicluster-data/master-0/data/instance:
uuid: "59c22a73e63f4545a7f9b8b1e2ebd3a7"
format_stamp: "Formatted at 2026-06-19 09:01:52 on dist-test-slave-mtdr"
server_key: "fb47d3b3bf1d43e3114ce13c3e46a89c"
server_key_iv: "cb4166ed1f100a64085d8fa254299c92"
server_key_version: "encryptionkey@0"
I20260619 09:01:52.529145 10112 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781859658134510-392-0/minicluster-data/master-0/wal/instance:
uuid: "59c22a73e63f4545a7f9b8b1e2ebd3a7"
format_stamp: "Formatted at 2026-06-19 09:01:52 on dist-test-slave-mtdr"
server_key: "fb47d3b3bf1d43e3114ce13c3e46a89c"
server_key_iv: "cb4166ed1f100a64085d8fa254299c92"
server_key_version: "encryptionkey@0"
I20260619 09:01:52.530239 10112 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 09:01:52.530856 10126 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 09:01:52.530992 10112 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 09:01:52.531069 10112 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781859658134510-392-0/minicluster-data/master-0/data,/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781859658134510-392-0/minicluster-data/master-0/wal
uuid: "59c22a73e63f4545a7f9b8b1e2ebd3a7"
format_stamp: "Formatted at 2026-06-19 09:01:52 on dist-test-slave-mtdr"
server_key: "fb47d3b3bf1d43e3114ce13c3e46a89c"
server_key_iv: "cb4166ed1f100a64085d8fa254299c92"
server_key_version: "encryptionkey@0"
I20260619 09:01:52.531128 10112 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781859658134510-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781859658134510-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781859658134510-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 09:01:52.534162 10112 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 09:01:52.534379 10112 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 09:01:52.534499 10112 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 09:01:52.537882 10112 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34151
I20260619 09:01:52.537916 10178 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34151 every 8 connection(s)
I20260619 09:01:52.538177 10112 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781859658134510-392-0/minicluster-data/master-0/data/info.pb
I20260619 09:01:52.538730 10179 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 09:01:52.541173 10179 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 59c22a73e63f4545a7f9b8b1e2ebd3a7: Bootstrap starting.
I20260619 09:01:52.541648 10179 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 59c22a73e63f4545a7f9b8b1e2ebd3a7: Neither blocks nor log segments found. Creating new log.
I20260619 09:01:52.541862 10179 log.cc:824] T 00000000000000000000000000000000 P 59c22a73e63f4545a7f9b8b1e2ebd3a7: Log is configured to *not* fsync() on all Append() calls
I20260619 09:01:52.542419 10179 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 59c22a73e63f4545a7f9b8b1e2ebd3a7: No bootstrap required, opened a new log
I20260619 09:01:52.543903 10179 raft_consensus.cc:348] T 00000000000000000000000000000000 P 59c22a73e63f4545a7f9b8b1e2ebd3a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59c22a73e63f4545a7f9b8b1e2ebd3a7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34151 } }
I20260619 09:01:52.544023 10179 raft_consensus.cc:374] T 00000000000000000000000000000000 P 59c22a73e63f4545a7f9b8b1e2ebd3a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 09:01:52.544044 10179 raft_consensus.cc:729] T 00000000000000000000000000000000 P 59c22a73e63f4545a7f9b8b1e2ebd3a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59c22a73e63f4545a7f9b8b1e2ebd3a7, State: Initialized, Role: FOLLOWER
I20260619 09:01:52.544142 10179 consensus_queue.cc:260] T 00000000000000000000000000000000 P 59c22a73e63f4545a7f9b8b1e2ebd3a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59c22a73e63f4545a7f9b8b1e2ebd3a7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34151 } }
I20260619 09:01:52.544191 10179 raft_consensus.cc:388] T 00000000000000000000000000000000 P 59c22a73e63f4545a7f9b8b1e2ebd3a7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 09:01:52.544219 10179 raft_consensus.cc:482] T 00000000000000000000000000000000 P 59c22a73e63f4545a7f9b8b1e2ebd3a7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 09:01:52.544261 10179 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 59c22a73e63f4545a7f9b8b1e2ebd3a7 [term 0 FOLLOWER]: Advancing to term 1
I20260619 09:01:52.544310 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ez5NB/build/release/bin/kudu as pid 10112
I20260619 09:01:52.544402 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781859658134510-392-0/minicluster-data/master-0/wal/instance
I20260619 09:01:52.544575 392 external_mini_cluster.cc:1428] Setting key d16df999953769c93b66cb16146c82b6
I20260619 09:01:52.544921 10179 raft_consensus.cc:504] T 00000000000000000000000000000000 P 59c22a73e63f4545a7f9b8b1e2ebd3a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59c22a73e63f4545a7f9b8b1e2ebd3a7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34151 } }
I20260619 09:01:52.545025 10179 leader_election.cc:302] T 00000000000000000000000000000000 P 59c22a73e63f4545a7f9b8b1e2ebd3a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 59c22a73e63f4545a7f9b8b1e2ebd3a7; no voters:
I20260619 09:01:52.545199 10179 leader_election.cc:288] T 00000000000000000000000000000000 P 59c22a73e63f4545a7f9b8b1e2ebd3a7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 09:01:52.545380 10185 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 59c22a73e63f4545a7f9b8b1e2ebd3a7 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 09:01:52.545455 10179 sys_catalog.cc:564] T 00000000000000000000000000000000 P 59c22a73e63f4545a7f9b8b1e2ebd3a7 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 09:01:52.545507 10185 raft_consensus.cc:686] T 00000000000000000000000000000000 P 59c22a73e63f4545a7f9b8b1e2ebd3a7 [term 1 LEADER]: Becoming Leader. State: Replica: 59c22a73e63f4545a7f9b8b1e2ebd3a7, State: Running, Role: LEADER
I20260619 09:01:52.545626 10185 consensus_queue.cc:237] T 00000000000000000000000000000000 P 59c22a73e63f4545a7f9b8b1e2ebd3a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59c22a73e63f4545a7f9b8b1e2ebd3a7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34151 } }
I20260619 09:01:52.546238 10185 sys_catalog.cc:455] T 00000000000000000000000000000000 P 59c22a73e63f4545a7f9b8b1e2ebd3a7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 59c22a73e63f4545a7f9b8b1e2ebd3a7. Latest consensus state: current_term: 1 leader_uuid: "59c22a73e63f4545a7f9b8b1e2ebd3a7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59c22a73e63f4545a7f9b8b1e2ebd3a7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34151 } } }
I20260619 09:01:52.546403 10185 sys_catalog.cc:458] T 00000000000000000000000000000000 P 59c22a73e63f4545a7f9b8b1e2ebd3a7 [sys.catalog]: This master's current role is: LEADER
I20260619 09:01:52.546243 10184 sys_catalog.cc:455] T 00000000000000000000000000000000 P 59c22a73e63f4545a7f9b8b1e2ebd3a7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "59c22a73e63f4545a7f9b8b1e2ebd3a7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59c22a73e63f4545a7f9b8b1e2ebd3a7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34151 } } }
I20260619 09:01:52.546685 10184 sys_catalog.cc:458] T 00000000000000000000000000000000 P 59c22a73e63f4545a7f9b8b1e2ebd3a7 [sys.catalog]: This master's current role is: LEADER
I20260619 09:01:52.547132 10183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:52.545060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54140 (local address 127.0.98.62:34151)
0619 09:01:52.545286 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:52.545294 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:52.545331 (+ 37us) server_negotiation.cc:407] Connection header received
0619 09:01:52.545436 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:52.545443 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:52.545484 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:52.545616 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:01:52.546622 (+ 1006us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:01:52.546625 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:01:52.546626 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 09:01:52.546630 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:01:52.546683 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 09:01:52.546698 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:01:52.546753 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0619 09:01:52.546892 (+ 139us) server_negotiation.cc:299] Negotiation successful
0619 09:01:52.546948 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":105,"threads_started":1}
W20260619 09:01:52.547171 10199 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 59c22a73e63f4545a7f9b8b1e2ebd3a7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 09:01:52.547351 10199 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 09:01:52.547410 10197 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 09:01:52.547524 10197 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 09:01:52.551713 10197 catalog_manager.cc:1347] Generated new cluster ID: 188500267ea044769384f1fe36cf6795
I20260619 09:01:52.551753 10197 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 09:01:52.574741 10197 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 09:01:52.574916 10197 catalog_manager.cc:1504] Loading token signing keys...
I20260619 09:01:52.577394 10197 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 59c22a73e63f4545a7f9b8b1e2ebd3a7: Generated new TSK 0
I20260619 09:01:52.577553 10197 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 09:01:52.581027 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ez5NB/build/release/bin/kudu
/tmp/dist-test-task4ez5NB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781859658134510-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781859658134510-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781859658134510-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781859658134510-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34151
--builtin_ntp_servers=127.0.98.20:36495
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260619 09:01:52.658951 10203 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 09:01:52.659138 10203 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 09:01:52.659166 10203 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 09:01:52.660284 10203 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 09:01:52.660320 10203 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 09:01:52.660352 10203 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 09:01:52.661625 10203 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36495
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781859658134510-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781859658134510-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781859658134510-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34151
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10203
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781859658134510-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 09:01:52.661892 10203 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 09:01:52.662149 10203 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 09:01:52.666771 10208 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:01:52.666759 10209 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:01:52.666926 10203 server_base.cc:1034] running on GCE node
W20260619 09:01:52.667083 10211 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:01:52.667302 10203 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 09:01:52.667536 10203 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 09:01:52.668675 10203 hybrid_clock.cc:648] HybridClock initialized: now 1781859712668652 us; error 39 us; skew 500 ppm
I20260619 09:01:52.669811 10203 webserver.cc:460] Webserver started at http://127.0.98.1:41125/ using document root <none> and password file <none>
I20260619 09:01:52.670015 10203 fs_manager.cc:359] Metadata directory not provided
I20260619 09:01:52.670061 10203 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 09:01:52.670195 10203 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 09:01:52.671233 10203 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781859658134510-392-0/minicluster-data/ts-0/data/instance:
uuid: "75f0b8ab7e0d40ed96030d4aa8dedadc"
format_stamp: "Formatted at 2026-06-19 09:01:52 on dist-test-slave-mtdr"
server_key: "2db81d6818a99d2f0c2b57710885e0c3"
server_key_iv: "6e51f8e16373844f1c5fb4b3c2a69a2b"
server_key_version: "encryptionkey@0"
I20260619 09:01:52.671538 10203 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781859658134510-392-0/minicluster-data/ts-0/wal/instance:
uuid: "75f0b8ab7e0d40ed96030d4aa8dedadc"
format_stamp: "Formatted at 2026-06-19 09:01:52 on dist-test-slave-mtdr"
server_key: "2db81d6818a99d2f0c2b57710885e0c3"
server_key_iv: "6e51f8e16373844f1c5fb4b3c2a69a2b"
server_key_version: "encryptionkey@0"
I20260619 09:01:52.672676 10203 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 09:01:52.673316 10217 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 09:01:52.673461 10203 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 09:01:52.673535 10203 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781859658134510-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781859658134510-392-0/minicluster-data/ts-0/wal
uuid: "75f0b8ab7e0d40ed96030d4aa8dedadc"
format_stamp: "Formatted at 2026-06-19 09:01:52 on dist-test-slave-mtdr"
server_key: "2db81d6818a99d2f0c2b57710885e0c3"
server_key_iv: "6e51f8e16373844f1c5fb4b3c2a69a2b"
server_key_version: "encryptionkey@0"
I20260619 09:01:52.673591 10203 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781859658134510-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781859658134510-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781859658134510-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 09:01:52.676752 10203 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 09:01:52.676970 10203 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 09:01:52.677079 10203 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 09:01:52.677260 10203 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 09:01:52.677515 10203 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 09:01:52.677546 10203 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 09:01:52.677570 10203 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 09:01:52.677593 10203 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 09:01:52.682663 10203 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34221
I20260619 09:01:52.682734 10330 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34221 every 8 connection(s)
I20260619 09:01:52.682997 10203 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781859658134510-392-0/minicluster-data/ts-0/data/info.pb
I20260619 09:01:52.685333 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ez5NB/build/release/bin/kudu as pid 10203
I20260619 09:01:52.685425 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781859658134510-392-0/minicluster-data/ts-0/wal/instance
I20260619 09:01:52.685631 392 external_mini_cluster.cc:1428] Setting key 079237423283b70526017d5b22afcae9
I20260619 09:01:52.685798 10183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:52.683647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53843 (local address 127.0.98.62:34151)
0619 09:01:52.683691 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:52.683694 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:52.683913 (+ 219us) server_negotiation.cc:407] Connection header received
0619 09:01:52.684339 (+ 426us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:52.684340 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:52.684346 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:52.684383 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:01:52.685541 (+ 1158us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:01:52.685543 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:01:52.685543 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 09:01:52.685549 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:01:52.685564 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 09:01:52.685568 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:01:52.685615 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0619 09:01:52.685716 (+ 101us) server_negotiation.cc:299] Negotiation successful
0619 09:01:52.685731 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260619 09:01:52.685966 10333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:52.683626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34151 (local address 127.0.98.1:53843)
0619 09:01:52.683808 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0619 09:01:52.683838 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0619 09:01:52.684254 (+ 416us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:01:52.684450 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:01:52.684452 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:01:52.684460 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0619 09:01:52.685339 (+ 879us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 09:01:52.685340 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 09:01:52.685376 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 09:01:52.685412 (+ 36us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 09:01:52.685628 (+ 216us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 09:01:52.685633 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 09:01:52.685636 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 09:01:52.685711 (+ 75us) client_negotiation.cc:240] Negotiation successful
0619 09:01:52.685781 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":86,"threads_started":1}
I20260619 09:01:52.686477 10331 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34151
I20260619 09:01:52.686566 10331 heartbeater.cc:464] Registering TS with master...
I20260619 09:01:52.686566 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ez5NB/build/release/bin/kudu
/tmp/dist-test-task4ez5NB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781859658134510-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781859658134510-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781859658134510-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781859658134510-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34151
--builtin_ntp_servers=127.0.98.20:36495
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260619 09:01:52.686707 10331 heartbeater.cc:511] Master 127.0.98.62:34151 requested a full tablet report, sending...
I20260619 09:01:52.687186 10143 ts_manager.cc:194] Registered new tserver with Master: 75f0b8ab7e0d40ed96030d4aa8dedadc (127.0.98.1:34221)
W20260619 09:01:52.763576 10334 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 09:01:52.763753 10334 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 09:01:52.763773 10334 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 09:01:52.764886 10334 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 09:01:52.764922 10334 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 09:01:52.764945 10334 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 09:01:52.766237 10334 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36495
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781859658134510-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781859658134510-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781859658134510-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34151
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10334
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781859658134510-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 09:01:52.766438 10334 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 09:01:52.766692 10334 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 09:01:52.771266 10340 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:01:52.771253 10342 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:01:52.771282 10339 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:01:52.771377 10334 server_base.cc:1034] running on GCE node
I20260619 09:01:52.771592 10334 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 09:01:52.771778 10334 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 09:01:52.772934 10334 hybrid_clock.cc:648] HybridClock initialized: now 1781859712772921 us; error 25 us; skew 500 ppm
I20260619 09:01:52.774199 10334 webserver.cc:460] Webserver started at http://127.0.98.2:37419/ using document root <none> and password file <none>
I20260619 09:01:52.774430 10334 fs_manager.cc:359] Metadata directory not provided
I20260619 09:01:52.774478 10334 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 09:01:52.774605 10334 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 09:01:52.775728 10334 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781859658134510-392-0/minicluster-data/ts-1/data/instance:
uuid: "931a45b0eaf24a438ba72d5a84048cb9"
format_stamp: "Formatted at 2026-06-19 09:01:52 on dist-test-slave-mtdr"
server_key: "d47fad4082667a42ceacf77cbf7c26ad"
server_key_iv: "2e6bd6fe669076aafe995fbb467b6ddd"
server_key_version: "encryptionkey@0"
I20260619 09:01:52.776115 10334 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781859658134510-392-0/minicluster-data/ts-1/wal/instance:
uuid: "931a45b0eaf24a438ba72d5a84048cb9"
format_stamp: "Formatted at 2026-06-19 09:01:52 on dist-test-slave-mtdr"
server_key: "d47fad4082667a42ceacf77cbf7c26ad"
server_key_iv: "2e6bd6fe669076aafe995fbb467b6ddd"
server_key_version: "encryptionkey@0"
I20260619 09:01:52.777406 10334 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260619 09:01:52.778127 10348 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 09:01:52.778281 10334 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 09:01:52.778354 10334 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781859658134510-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781859658134510-392-0/minicluster-data/ts-1/wal
uuid: "931a45b0eaf24a438ba72d5a84048cb9"
format_stamp: "Formatted at 2026-06-19 09:01:52 on dist-test-slave-mtdr"
server_key: "d47fad4082667a42ceacf77cbf7c26ad"
server_key_iv: "2e6bd6fe669076aafe995fbb467b6ddd"
server_key_version: "encryptionkey@0"
I20260619 09:01:52.778425 10334 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781859658134510-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781859658134510-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781859658134510-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 09:01:52.781754 10334 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 09:01:52.781986 10334 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 09:01:52.782110 10334 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 09:01:52.782305 10334 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 09:01:52.782582 10334 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 09:01:52.782616 10334 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 09:01:52.782646 10334 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 09:01:52.782672 10334 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 09:01:52.788520 10334 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:46619
I20260619 09:01:52.788550 10461 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:46619 every 8 connection(s)
I20260619 09:01:52.788862 10334 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781859658134510-392-0/minicluster-data/ts-1/data/info.pb
I20260619 09:01:52.790746 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ez5NB/build/release/bin/kudu as pid 10334
I20260619 09:01:52.790851 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781859658134510-392-0/minicluster-data/ts-1/wal/instance
I20260619 09:01:52.791013 392 external_mini_cluster.cc:1428] Setting key fe55876aa84c5068e486dd5695560c87
I20260619 09:01:52.791849 10183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:52.789498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37843 (local address 127.0.98.62:34151)
0619 09:01:52.789555 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:52.789557 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:52.789746 (+ 189us) server_negotiation.cc:407] Connection header received
0619 09:01:52.790218 (+ 472us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:52.790219 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:52.790226 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:52.790288 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:01:52.791604 (+ 1316us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:01:52.791605 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:01:52.791606 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 09:01:52.791610 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:01:52.791627 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 09:01:52.791632 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:01:52.791680 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0619 09:01:52.791768 (+ 88us) server_negotiation.cc:299] Negotiation successful
0619 09:01:52.791782 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260619 09:01:52.791955 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ez5NB/build/release/bin/kudu
/tmp/dist-test-task4ez5NB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781859658134510-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781859658134510-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781859658134510-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781859658134510-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34151
--builtin_ntp_servers=127.0.98.20:36495
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260619 09:01:52.792007 10464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:52.789472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34151 (local address 127.0.98.2:37843)
0619 09:01:52.789646 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0619 09:01:52.789674 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0619 09:01:52.790124 (+ 450us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:01:52.790366 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:01:52.790369 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:01:52.790376 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0619 09:01:52.791436 (+ 1060us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 09:01:52.791436 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0619 09:01:52.791472 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 09:01:52.791506 (+ 34us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 09:01:52.791689 (+ 183us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 09:01:52.791694 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 09:01:52.791697 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 09:01:52.791775 (+ 78us) client_negotiation.cc:240] Negotiation successful
0619 09:01:52.791821 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":85,"threads_started":1}
I20260619 09:01:52.793272 10462 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34151
I20260619 09:01:52.793350 10462 heartbeater.cc:464] Registering TS with master...
I20260619 09:01:52.793478 10462 heartbeater.cc:511] Master 127.0.98.62:34151 requested a full tablet report, sending...
I20260619 09:01:52.793793 10143 ts_manager.cc:194] Registered new tserver with Master: 931a45b0eaf24a438ba72d5a84048cb9 (127.0.98.2:46619)
W20260619 09:01:52.868422 10465 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 09:01:52.868611 10465 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 09:01:52.868640 10465 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 09:01:52.869764 10465 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 09:01:52.869808 10465 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 09:01:52.869844 10465 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 09:01:52.871181 10465 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36495
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781859658134510-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781859658134510-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781859658134510-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34151
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10465
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781859658134510-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 09:01:52.871398 10465 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 09:01:52.871655 10465 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 09:01:52.876173 10470 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:01:52.876173 10473 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:01:52.876217 10471 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:01:52.876434 10465 server_base.cc:1034] running on GCE node
I20260619 09:01:52.876629 10465 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 09:01:52.876863 10465 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 09:01:52.878005 10465 hybrid_clock.cc:648] HybridClock initialized: now 1781859712877992 us; error 31 us; skew 500 ppm
I20260619 09:01:52.879236 10465 webserver.cc:460] Webserver started at http://127.0.98.3:35999/ using document root <none> and password file <none>
I20260619 09:01:52.879449 10465 fs_manager.cc:359] Metadata directory not provided
I20260619 09:01:52.879501 10465 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 09:01:52.879627 10465 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 09:01:52.880650 10465 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781859658134510-392-0/minicluster-data/ts-2/data/instance:
uuid: "a09f474357d642a2b6cf2ade05620052"
format_stamp: "Formatted at 2026-06-19 09:01:52 on dist-test-slave-mtdr"
server_key: "93f9f9fb54d73fd13139139b4adac1d8"
server_key_iv: "83003a283aae49909b714609b44d049e"
server_key_version: "encryptionkey@0"
I20260619 09:01:52.881039 10465 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781859658134510-392-0/minicluster-data/ts-2/wal/instance:
uuid: "a09f474357d642a2b6cf2ade05620052"
format_stamp: "Formatted at 2026-06-19 09:01:52 on dist-test-slave-mtdr"
server_key: "93f9f9fb54d73fd13139139b4adac1d8"
server_key_iv: "83003a283aae49909b714609b44d049e"
server_key_version: "encryptionkey@0"
I20260619 09:01:52.882246 10465 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 09:01:52.882936 10479 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 09:01:52.883090 10465 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260619 09:01:52.883172 10465 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781859658134510-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781859658134510-392-0/minicluster-data/ts-2/wal
uuid: "a09f474357d642a2b6cf2ade05620052"
format_stamp: "Formatted at 2026-06-19 09:01:52 on dist-test-slave-mtdr"
server_key: "93f9f9fb54d73fd13139139b4adac1d8"
server_key_iv: "83003a283aae49909b714609b44d049e"
server_key_version: "encryptionkey@0"
I20260619 09:01:52.883244 10465 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781859658134510-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781859658134510-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781859658134510-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 09:01:52.886540 10465 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 09:01:52.886785 10465 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 09:01:52.886958 10465 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 09:01:52.887149 10465 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 09:01:52.887440 10465 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 09:01:52.887475 10465 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 09:01:52.887523 10465 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 09:01:52.887552 10465 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 09:01:52.893126 10465 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37125
I20260619 09:01:52.893160 10592 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37125 every 8 connection(s)
I20260619 09:01:52.893440 10465 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781859658134510-392-0/minicluster-data/ts-2/data/info.pb
I20260619 09:01:52.896227 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ez5NB/build/release/bin/kudu as pid 10465
I20260619 09:01:52.896250 10183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:52.894074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60039 (local address 127.0.98.62:34151)
0619 09:01:52.894144 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:52.894146 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:52.894312 (+ 166us) server_negotiation.cc:407] Connection header received
0619 09:01:52.894762 (+ 450us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:52.894764 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:52.894769 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:52.894866 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:01:52.895966 (+ 1100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:01:52.895967 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:01:52.895968 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 09:01:52.895972 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:01:52.895986 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 09:01:52.895990 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:01:52.896053 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0619 09:01:52.896165 (+ 112us) server_negotiation.cc:299] Negotiation successful
0619 09:01:52.896181 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 09:01:52.896311 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781859658134510-392-0/minicluster-data/ts-2/wal/instance
I20260619 09:01:52.896459 10595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:52.894023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34151 (local address 127.0.98.3:60039)
0619 09:01:52.894187 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0619 09:01:52.894225 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 09:01:52.894657 (+ 432us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:01:52.894964 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:01:52.894968 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:01:52.894975 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0619 09:01:52.895798 (+ 823us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 09:01:52.895799 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 09:01:52.895833 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 09:01:52.895860 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 09:01:52.896078 (+ 218us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 09:01:52.896083 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 09:01:52.896086 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 09:01:52.896160 (+ 74us) client_negotiation.cc:240] Negotiation successful
0619 09:01:52.896208 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":73,"threads_started":1}
I20260619 09:01:52.896502 392 external_mini_cluster.cc:1428] Setting key b9d3d3d17efd15fb1b1339b160f0ebf2
I20260619 09:01:52.896876 10593 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34151
I20260619 09:01:52.896970 10593 heartbeater.cc:464] Registering TS with master...
I20260619 09:01:52.897114 10593 heartbeater.cc:511] Master 127.0.98.62:34151 requested a full tablet report, sending...
I20260619 09:01:52.897416 10142 ts_manager.cc:194] Registered new tserver with Master: a09f474357d642a2b6cf2ade05620052 (127.0.98.3:37125)
I20260619 09:01:52.908210 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260619 09:01:52.909950 10183 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 09:01:52.909537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54144 (local address 127.0.98.62:34151)
0619 09:01:52.909604 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:52.909605 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:52.909677 (+ 72us) server_negotiation.cc:407] Connection header received
0619 09:01:52.909718 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:52.909719 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:52.909723 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:52.909767 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:01:52.909883 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54144: BlockingRecv error: recv got EOF from 127.0.0.1:54144 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
I20260619 09:01:52.909984 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ez5NB/build/release/bin/kudu with pid 10203
I20260619 09:01:52.913880 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ez5NB/build/release/bin/kudu with pid 10334
I20260619 09:01:52.917466 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ez5NB/build/release/bin/kudu with pid 10465
I20260619 09:01:52.920925 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ez5NB/build/release/bin/kudu with pid 10112
2026-06-19T09:01:52Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (503 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781859658134510-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 09:01:52 dist-test-slave-mtdr krb5kdc[10603](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 09:01:52 dist-test-slave-mtdr krb5kdc[10603](info): set up 2 sockets
Jun 19 09:01:52 dist-test-slave-mtdr krb5kdc[10603](info): commencing operation
krb5kdc: starting...
W20260619 09:01:53.875573 392 mini_kdc.cc:121] Time spent starting KDC: real 0.937s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781859658134510-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781859658134510-392-0/krb5kdc/test-user.keytab.
Jun 19 09:01:53 dist-test-slave-mtdr krb5kdc[10603](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859713, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T09:01:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T09:01:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781859658134510-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781859658134510-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 09:01:53.983227 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ez5NB/build/release/bin/kudu
/tmp/dist-test-task4ez5NB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781859658134510-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781859658134510-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781859658134510-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781859658134510-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36633
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781859658134510-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43919
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36633
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781859658134510-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781859658134510-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781859658134510-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 09:01:54.059895 10616 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 09:01:54.060097 10616 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 09:01:54.060127 10616 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 09:01:54.061251 10616 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 09:01:54.061290 10616 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 09:01:54.061307 10616 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 09:01:54.061323 10616 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 09:01:54.061337 10616 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 09:01:54.062762 10616 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43919
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781859658134510-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781859658134510-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36633
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781859658134510-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36633
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781859658134510-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10616
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781859658134510-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 09:01:54.063014 10616 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 09:01:54.063215 10616 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 09:01:54.067776 10624 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:01:54.067852 10621 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:01:54.067816 10622 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:01:54.067899 10616 server_base.cc:1034] running on GCE node
I20260619 09:01:54.068167 10616 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 09:01:54.068447 10616 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 09:01:54.069591 10616 hybrid_clock.cc:648] HybridClock initialized: now 1781859714069579 us; error 27 us; skew 500 ppm
Jun 19 09:01:54 dist-test-slave-mtdr krb5kdc[10603](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859714, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 09:01:54.071977 10616 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 09:01:54.072376 10616 webserver.cc:460] Webserver started at http://127.0.98.62:42491/ using document root <none> and password file <none>
I20260619 09:01:54.072587 10616 fs_manager.cc:359] Metadata directory not provided
I20260619 09:01:54.072635 10616 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 09:01:54.072754 10616 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 09:01:54.073745 10616 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781859658134510-392-0/minicluster-data/master-0/data/instance:
uuid: "14322e8638334ab080cbeffd155ea099"
format_stamp: "Formatted at 2026-06-19 09:01:54 on dist-test-slave-mtdr"
server_key: "02d7f5c06a3df7f2b3c2b04fc0b0aadf"
server_key_iv: "e7d5d08536c7f657dec4bbe56a64f2cc"
server_key_version: "encryptionkey@0"
I20260619 09:01:54.074061 10616 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781859658134510-392-0/minicluster-data/master-0/wal/instance:
uuid: "14322e8638334ab080cbeffd155ea099"
format_stamp: "Formatted at 2026-06-19 09:01:54 on dist-test-slave-mtdr"
server_key: "02d7f5c06a3df7f2b3c2b04fc0b0aadf"
server_key_iv: "e7d5d08536c7f657dec4bbe56a64f2cc"
server_key_version: "encryptionkey@0"
I20260619 09:01:54.075299 10616 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 09:01:54.075959 10631 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 09:01:54.076107 10616 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 09:01:54.076186 10616 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781859658134510-392-0/minicluster-data/master-0/data,/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781859658134510-392-0/minicluster-data/master-0/wal
uuid: "14322e8638334ab080cbeffd155ea099"
format_stamp: "Formatted at 2026-06-19 09:01:54 on dist-test-slave-mtdr"
server_key: "02d7f5c06a3df7f2b3c2b04fc0b0aadf"
server_key_iv: "e7d5d08536c7f657dec4bbe56a64f2cc"
server_key_version: "encryptionkey@0"
I20260619 09:01:54.076243 10616 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781859658134510-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781859658134510-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781859658134510-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 09:01:54.111500 10616 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 09:01:54.113747 10616 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 09:01:54.113885 10616 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 09:01:54.117852 10616 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36633
I20260619 09:01:54.117880 10683 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36633 every 8 connection(s)
I20260619 09:01:54.118238 10616 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781859658134510-392-0/minicluster-data/master-0/data/info.pb
I20260619 09:01:54.119235 10684 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 09:01:54.121389 10684 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 14322e8638334ab080cbeffd155ea099: Bootstrap starting.
I20260619 09:01:54.121930 10684 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 14322e8638334ab080cbeffd155ea099: Neither blocks nor log segments found. Creating new log.
I20260619 09:01:54.122189 10684 log.cc:824] T 00000000000000000000000000000000 P 14322e8638334ab080cbeffd155ea099: Log is configured to *not* fsync() on all Append() calls
I20260619 09:01:54.122857 10684 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 14322e8638334ab080cbeffd155ea099: No bootstrap required, opened a new log
I20260619 09:01:54.124368 10684 raft_consensus.cc:348] T 00000000000000000000000000000000 P 14322e8638334ab080cbeffd155ea099 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14322e8638334ab080cbeffd155ea099" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36633 } }
I20260619 09:01:54.124500 10684 raft_consensus.cc:374] T 00000000000000000000000000000000 P 14322e8638334ab080cbeffd155ea099 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 09:01:54.124521 10684 raft_consensus.cc:729] T 00000000000000000000000000000000 P 14322e8638334ab080cbeffd155ea099 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14322e8638334ab080cbeffd155ea099, State: Initialized, Role: FOLLOWER
I20260619 09:01:54.124653 10684 consensus_queue.cc:260] T 00000000000000000000000000000000 P 14322e8638334ab080cbeffd155ea099 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14322e8638334ab080cbeffd155ea099" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36633 } }
I20260619 09:01:54.124722 10684 raft_consensus.cc:388] T 00000000000000000000000000000000 P 14322e8638334ab080cbeffd155ea099 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 09:01:54.124743 10684 raft_consensus.cc:482] T 00000000000000000000000000000000 P 14322e8638334ab080cbeffd155ea099 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 09:01:54.124786 10684 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 14322e8638334ab080cbeffd155ea099 [term 0 FOLLOWER]: Advancing to term 1
I20260619 09:01:54.125504 10684 raft_consensus.cc:504] T 00000000000000000000000000000000 P 14322e8638334ab080cbeffd155ea099 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14322e8638334ab080cbeffd155ea099" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36633 } }
I20260619 09:01:54.125591 10684 leader_election.cc:302] T 00000000000000000000000000000000 P 14322e8638334ab080cbeffd155ea099 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 14322e8638334ab080cbeffd155ea099; no voters:
I20260619 09:01:54.125734 10684 leader_election.cc:288] T 00000000000000000000000000000000 P 14322e8638334ab080cbeffd155ea099 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 09:01:54.125780 10687 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 14322e8638334ab080cbeffd155ea099 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 09:01:54.125926 10687 raft_consensus.cc:686] T 00000000000000000000000000000000 P 14322e8638334ab080cbeffd155ea099 [term 1 LEADER]: Becoming Leader. State: Replica: 14322e8638334ab080cbeffd155ea099, State: Running, Role: LEADER
I20260619 09:01:54.125972 10684 sys_catalog.cc:564] T 00000000000000000000000000000000 P 14322e8638334ab080cbeffd155ea099 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 09:01:54.126034 10687 consensus_queue.cc:237] T 00000000000000000000000000000000 P 14322e8638334ab080cbeffd155ea099 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14322e8638334ab080cbeffd155ea099" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36633 } }
I20260619 09:01:54.126385 10688 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14322e8638334ab080cbeffd155ea099 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "14322e8638334ab080cbeffd155ea099" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14322e8638334ab080cbeffd155ea099" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36633 } } }
I20260619 09:01:54.126410 10689 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14322e8638334ab080cbeffd155ea099 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 14322e8638334ab080cbeffd155ea099. Latest consensus state: current_term: 1 leader_uuid: "14322e8638334ab080cbeffd155ea099" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14322e8638334ab080cbeffd155ea099" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36633 } } }
I20260619 09:01:54.126495 10688 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14322e8638334ab080cbeffd155ea099 [sys.catalog]: This master's current role is: LEADER
I20260619 09:01:54.126753 10689 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14322e8638334ab080cbeffd155ea099 [sys.catalog]: This master's current role is: LEADER
I20260619 09:01:54.126762 10696 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 09:01:54.127228 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ez5NB/build/release/bin/kudu as pid 10616
I20260619 09:01:54.127279 10696 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 09:01:54.127316 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781859658134510-392-0/minicluster-data/master-0/wal/instance
I20260619 09:01:54.127506 392 external_mini_cluster.cc:1428] Setting key 28fddfea4017ddd899e89a65ea9a80f5
I20260619 09:01:54.128757 10696 catalog_manager.cc:1347] Generated new cluster ID: 97463db618734932adfc61cf080d9cc9
I20260619 09:01:54.128803 10696 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 19 09:01:54 dist-test-slave-mtdr krb5kdc[10603](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859713, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 09:01:54.149206 10706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:54.128231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59574 (local address 127.0.98.62:36633)
0619 09:01:54.128380 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:54.128389 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:54.128420 (+ 31us) server_negotiation.cc:407] Connection header received
0619 09:01:54.128540 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:54.128547 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:54.128612 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:54.138916 (+ 10304us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:01:54.139680 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:54.141015 (+ 1335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:54.142064 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:54.142648 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:54.142734 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:54.145387 (+ 2653us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:01:54.145393 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:01:54.145393 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:01:54.145405 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:01:54.147299 (+ 1894us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:01:54.147648 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:01:54.147651 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:01:54.147651 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:01:54.147755 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:01:54.147919 (+ 164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:01:54.147920 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:01:54.147920 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:01:54.148232 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:01:54.148753 (+ 521us) server_negotiation.cc:1035] Waiting for connection context
0619 09:01:54.148822 (+ 69us) server_negotiation.cc:299] Negotiation successful
0619 09:01:54.148956 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":72,"threads_started":1}
I20260619 09:01:54.163558 10696 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 09:01:54.164093 10696 catalog_manager.cc:1504] Loading token signing keys...
I20260619 09:01:54.168224 10696 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 14322e8638334ab080cbeffd155ea099: Generated new TSK 0
I20260619 09:01:54.168406 10696 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781859658134510-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781859658134510-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 09:01:54.197149 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ez5NB/build/release/bin/kudu
/tmp/dist-test-task4ez5NB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781859658134510-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781859658134510-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781859658134510-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781859658134510-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36633
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781859658134510-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43919
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781859658134510-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781859658134510-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781859658134510-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 09:01:54.275319 10710 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 09:01:54.275501 10710 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 09:01:54.275522 10710 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 09:01:54.276615 10710 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 09:01:54.276654 10710 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 09:01:54.276677 10710 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 09:01:54.278070 10710 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43919
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781859658134510-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781859658134510-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781859658134510-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781859658134510-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36633
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10710
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781859658134510-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 09:01:54.278276 10710 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 09:01:54.278475 10710 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 09:01:54.283185 10716 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:01:54.283257 10718 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:01:54.283209 10715 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:01:54.283278 10710 server_base.cc:1034] running on GCE node
I20260619 09:01:54.283540 10710 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 09:01:54.283739 10710 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 09:01:54.284863 10710 hybrid_clock.cc:648] HybridClock initialized: now 1781859714284855 us; error 29 us; skew 500 ppm
Jun 19 09:01:54 dist-test-slave-mtdr krb5kdc[10603](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859714, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 09:01:54.287096 10710 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 09:01:54.287480 10710 webserver.cc:460] Webserver started at http://127.0.98.1:44503/ using document root <none> and password file <none>
I20260619 09:01:54.287684 10710 fs_manager.cc:359] Metadata directory not provided
I20260619 09:01:54.287729 10710 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 09:01:54.287827 10710 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 09:01:54.288810 10710 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781859658134510-392-0/minicluster-data/ts-0/data/instance:
uuid: "f057e8287b944303bb8a54ab05d3f496"
format_stamp: "Formatted at 2026-06-19 09:01:54 on dist-test-slave-mtdr"
server_key: "a6821819942e3f664dbb11ebd123586c"
server_key_iv: "3667a501277b7eb8a5d9239c2ee63466"
server_key_version: "encryptionkey@0"
I20260619 09:01:54.289115 10710 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781859658134510-392-0/minicluster-data/ts-0/wal/instance:
uuid: "f057e8287b944303bb8a54ab05d3f496"
format_stamp: "Formatted at 2026-06-19 09:01:54 on dist-test-slave-mtdr"
server_key: "a6821819942e3f664dbb11ebd123586c"
server_key_iv: "3667a501277b7eb8a5d9239c2ee63466"
server_key_version: "encryptionkey@0"
I20260619 09:01:54.290292 10710 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 09:01:54.290989 10725 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 09:01:54.291115 10710 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 09:01:54.291184 10710 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781859658134510-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781859658134510-392-0/minicluster-data/ts-0/wal
uuid: "f057e8287b944303bb8a54ab05d3f496"
format_stamp: "Formatted at 2026-06-19 09:01:54 on dist-test-slave-mtdr"
server_key: "a6821819942e3f664dbb11ebd123586c"
server_key_iv: "3667a501277b7eb8a5d9239c2ee63466"
server_key_version: "encryptionkey@0"
I20260619 09:01:54.291249 10710 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781859658134510-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781859658134510-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781859658134510-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 09:01:54.316536 10710 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 09:01:54.318362 10710 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 09:01:54.318475 10710 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 09:01:54.318710 10710 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 09:01:54.319036 10710 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 09:01:54.319069 10710 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 09:01:54.319089 10710 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 09:01:54.319111 10710 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 09:01:54.324400 10710 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41969
I20260619 09:01:54.324446 10838 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41969 every 8 connection(s)
I20260619 09:01:54.324715 10710 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781859658134510-392-0/minicluster-data/ts-0/data/info.pb
Jun 19 09:01:54 dist-test-slave-mtdr krb5kdc[10603](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859714, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 09:01:54.331450 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ez5NB/build/release/bin/kudu as pid 10710
I20260619 09:01:54.331548 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781859658134510-392-0/minicluster-data/ts-0/wal/instance
I20260619 09:01:54.331722 392 external_mini_cluster.cc:1428] Setting key 8ca83233be04154c67913bc1fb097246
I20260619 09:01:54.333416 10706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:54.325349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50303 (local address 127.0.98.62:36633)
0619 09:01:54.325412 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:54.325413 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:54.325603 (+ 190us) server_negotiation.cc:407] Connection header received
0619 09:01:54.326077 (+ 474us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:54.326078 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:54.326089 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:54.326141 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:01:54.327109 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:54.327654 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:54.328603 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:54.328802 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:54.328858 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:54.330720 (+ 1862us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:01:54.330722 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:01:54.330722 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:01:54.330730 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:01:54.332106 (+ 1376us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:01:54.332512 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:01:54.332514 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:01:54.332514 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:01:54.332570 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:01:54.332813 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:01:54.332815 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:01:54.332815 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:01:54.332926 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:01:54.332981 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0619 09:01:54.333242 (+ 261us) server_negotiation.cc:299] Negotiation successful
0619 09:01:54.333327 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260619 09:01:54.333613 10841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:54.325317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36633 (local address 127.0.98.1:50303)
0619 09:01:54.325484 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0619 09:01:54.325523 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 09:01:54.325998 (+ 475us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:01:54.326196 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:01:54.326199 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:01:54.326778 (+ 579us) client_negotiation.cc:189] Negotiated authn=SASL
0619 09:01:54.327013 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:54.327019 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:54.327727 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:54.327728 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:54.328523 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:54.328525 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:54.328866 (+ 341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:54.328868 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:54.328986 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:54.329399 (+ 413us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 09:01:54.329406 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0619 09:01:54.330632 (+ 1226us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 09:01:54.332184 (+ 1552us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:01:54.332190 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:01:54.332192 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:01:54.332448 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:01:54.332657 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:01:54.332658 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:01:54.332659 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:01:54.332755 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:01:54.332979 (+ 224us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 09:01:54.332984 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 09:01:54.333113 (+ 129us) client_negotiation.cc:769] Sending connection context
0619 09:01:54.333209 (+ 96us) client_negotiation.cc:240] Negotiation successful
0619 09:01:54.333368 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":83,"threads_started":1}
I20260619 09:01:54.334049 10839 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36633
I20260619 09:01:54.334129 10839 heartbeater.cc:464] Registering TS with master...
I20260619 09:01:54.334287 10839 heartbeater.cc:511] Master 127.0.98.62:36633 requested a full tablet report, sending...
I20260619 09:01:54.334743 10648 ts_manager.cc:194] Registered new tserver with Master: f057e8287b944303bb8a54ab05d3f496 (127.0.98.1:41969)
I20260619 09:01:54.335347 10648 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:50303
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781859658134510-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781859658134510-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 09:01:54.353878 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ez5NB/build/release/bin/kudu
/tmp/dist-test-task4ez5NB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781859658134510-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781859658134510-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781859658134510-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781859658134510-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36633
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781859658134510-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43919
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781859658134510-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781859658134510-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781859658134510-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 09:01:54.430475 10844 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 09:01:54.430658 10844 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 09:01:54.430680 10844 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 09:01:54.431818 10844 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 09:01:54.431856 10844 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 09:01:54.431881 10844 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 09:01:54.433270 10844 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43919
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781859658134510-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781859658134510-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781859658134510-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781859658134510-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36633
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10844
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781859658134510-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 09:01:54.433470 10844 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 09:01:54.433683 10844 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 09:01:54.438138 10852 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:01:54.438145 10849 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:01:54.438146 10850 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:01:54.438493 10844 server_base.cc:1034] running on GCE node
I20260619 09:01:54.438664 10844 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 09:01:54.438930 10844 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 09:01:54.440076 10844 hybrid_clock.cc:648] HybridClock initialized: now 1781859714440060 us; error 31 us; skew 500 ppm
Jun 19 09:01:54 dist-test-slave-mtdr krb5kdc[10603](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859714, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 09:01:54.442414 10844 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 09:01:54.442790 10844 webserver.cc:460] Webserver started at http://127.0.98.2:32887/ using document root <none> and password file <none>
I20260619 09:01:54.443019 10844 fs_manager.cc:359] Metadata directory not provided
I20260619 09:01:54.443069 10844 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 09:01:54.443183 10844 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 09:01:54.444161 10844 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781859658134510-392-0/minicluster-data/ts-1/data/instance:
uuid: "ae32eaeaae174261ab3a2d65409f77d9"
format_stamp: "Formatted at 2026-06-19 09:01:54 on dist-test-slave-mtdr"
server_key: "372d430df105ee539d6e4f8d44e18595"
server_key_iv: "2209a8d5e3001cf1d6ed5a97abd33f98"
server_key_version: "encryptionkey@0"
I20260619 09:01:54.444484 10844 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781859658134510-392-0/minicluster-data/ts-1/wal/instance:
uuid: "ae32eaeaae174261ab3a2d65409f77d9"
format_stamp: "Formatted at 2026-06-19 09:01:54 on dist-test-slave-mtdr"
server_key: "372d430df105ee539d6e4f8d44e18595"
server_key_iv: "2209a8d5e3001cf1d6ed5a97abd33f98"
server_key_version: "encryptionkey@0"
I20260619 09:01:54.445680 10844 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 09:01:54.446362 10859 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 09:01:54.446517 10844 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 09:01:54.446594 10844 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781859658134510-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781859658134510-392-0/minicluster-data/ts-1/wal
uuid: "ae32eaeaae174261ab3a2d65409f77d9"
format_stamp: "Formatted at 2026-06-19 09:01:54 on dist-test-slave-mtdr"
server_key: "372d430df105ee539d6e4f8d44e18595"
server_key_iv: "2209a8d5e3001cf1d6ed5a97abd33f98"
server_key_version: "encryptionkey@0"
I20260619 09:01:54.446651 10844 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781859658134510-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781859658134510-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781859658134510-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 09:01:54.456439 10844 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 09:01:54.458176 10844 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 09:01:54.458302 10844 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 09:01:54.458496 10844 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 09:01:54.458760 10844 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 09:01:54.458791 10844 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 09:01:54.458866 10844 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 09:01:54.458894 10844 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 09:01:54.464010 10844 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39833
I20260619 09:01:54.464049 10972 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39833 every 8 connection(s)
I20260619 09:01:54.464319 10844 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781859658134510-392-0/minicluster-data/ts-1/data/info.pb
I20260619 09:01:54.467886 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ez5NB/build/release/bin/kudu as pid 10844
I20260619 09:01:54.467980 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781859658134510-392-0/minicluster-data/ts-1/wal/instance
I20260619 09:01:54.468168 392 external_mini_cluster.cc:1428] Setting key 1d076927db2fc479b74465a76ecbafbf
Jun 19 09:01:54 dist-test-slave-mtdr krb5kdc[10603](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859714, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 09:01:54.473493 10706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:54.464977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42127 (local address 127.0.98.62:36633)
0619 09:01:54.465025 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:54.465027 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:54.465219 (+ 192us) server_negotiation.cc:407] Connection header received
0619 09:01:54.465678 (+ 459us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:54.465679 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:54.465687 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:54.465727 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:01:54.466642 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:54.467202 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:54.468378 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:54.468607 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:54.468670 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:54.470748 (+ 2078us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:01:54.470749 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:01:54.470749 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:01:54.470756 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:01:54.472041 (+ 1285us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:01:54.472523 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:01:54.472524 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:01:54.472524 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:01:54.472581 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:01:54.472852 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:01:54.472852 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:01:54.472853 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:01:54.472991 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:01:54.473051 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0619 09:01:54.473362 (+ 311us) server_negotiation.cc:299] Negotiation successful
0619 09:01:54.473430 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260619 09:01:54.473721 10975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:54.464954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36633 (local address 127.0.98.2:42127)
0619 09:01:54.465107 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0619 09:01:54.465147 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 09:01:54.465590 (+ 443us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:01:54.465793 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:01:54.465796 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:01:54.466374 (+ 578us) client_negotiation.cc:189] Negotiated authn=SASL
0619 09:01:54.466560 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:54.466564 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:54.467279 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:54.467280 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:54.468300 (+ 1020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:54.468303 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:54.468684 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:54.468685 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:54.468816 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:54.469258 (+ 442us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 09:01:54.469266 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 09:01:54.470648 (+ 1382us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 09:01:54.472113 (+ 1465us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:01:54.472119 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:01:54.472122 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:01:54.472426 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:01:54.472651 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:01:54.472652 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:01:54.472652 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:01:54.472783 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:01:54.473060 (+ 277us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 09:01:54.473065 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 09:01:54.473223 (+ 158us) client_negotiation.cc:769] Sending connection context
0619 09:01:54.473345 (+ 122us) client_negotiation.cc:240] Negotiation successful
0619 09:01:54.473504 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":67,"threads_started":1}
I20260619 09:01:54.474290 10973 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36633
I20260619 09:01:54.474376 10973 heartbeater.cc:464] Registering TS with master...
I20260619 09:01:54.474547 10973 heartbeater.cc:511] Master 127.0.98.62:36633 requested a full tablet report, sending...
I20260619 09:01:54.474931 10648 ts_manager.cc:194] Registered new tserver with Master: ae32eaeaae174261ab3a2d65409f77d9 (127.0.98.2:39833)
I20260619 09:01:54.475427 10648 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:42127
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781859658134510-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781859658134510-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 09:01:54.491034 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ez5NB/build/release/bin/kudu
/tmp/dist-test-task4ez5NB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781859658134510-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781859658134510-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781859658134510-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781859658134510-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36633
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781859658134510-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43919
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781859658134510-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781859658134510-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781859658134510-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 09:01:54.567891 10978 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 09:01:54.568081 10978 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 09:01:54.568104 10978 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 09:01:54.569227 10978 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 09:01:54.569265 10978 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 09:01:54.569289 10978 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 09:01:54.570655 10978 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43919
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781859658134510-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781859658134510-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781859658134510-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781859658134510-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36633
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10978
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781859658134510-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 09:01:54.570894 10978 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 09:01:54.571094 10978 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 09:01:54.575719 10983 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:01:54.575740 10984 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:01:54.575798 10978 server_base.cc:1034] running on GCE node
W20260619 09:01:54.575704 10986 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:01:54.576097 10978 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 09:01:54.576309 10978 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 09:01:54.577452 10978 hybrid_clock.cc:648] HybridClock initialized: now 1781859714577437 us; error 28 us; skew 500 ppm
Jun 19 09:01:54 dist-test-slave-mtdr krb5kdc[10603](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859714, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 09:01:54.579799 10978 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 09:01:54.580209 10978 webserver.cc:460] Webserver started at http://127.0.98.3:46015/ using document root <none> and password file <none>
I20260619 09:01:54.580401 10978 fs_manager.cc:359] Metadata directory not provided
I20260619 09:01:54.580447 10978 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 09:01:54.580564 10978 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 09:01:54.581539 10978 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781859658134510-392-0/minicluster-data/ts-2/data/instance:
uuid: "0d5b0c5586634312bf0ff0f501bfe75c"
format_stamp: "Formatted at 2026-06-19 09:01:54 on dist-test-slave-mtdr"
server_key: "274bce032385b8191f6cc6431f9c7162"
server_key_iv: "085bae559e1a299d193e154eb67986cd"
server_key_version: "encryptionkey@0"
I20260619 09:01:54.581848 10978 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781859658134510-392-0/minicluster-data/ts-2/wal/instance:
uuid: "0d5b0c5586634312bf0ff0f501bfe75c"
format_stamp: "Formatted at 2026-06-19 09:01:54 on dist-test-slave-mtdr"
server_key: "274bce032385b8191f6cc6431f9c7162"
server_key_iv: "085bae559e1a299d193e154eb67986cd"
server_key_version: "encryptionkey@0"
I20260619 09:01:54.583115 10978 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260619 09:01:54.583778 10993 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 09:01:54.583922 10978 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 09:01:54.584003 10978 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781859658134510-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781859658134510-392-0/minicluster-data/ts-2/wal
uuid: "0d5b0c5586634312bf0ff0f501bfe75c"
format_stamp: "Formatted at 2026-06-19 09:01:54 on dist-test-slave-mtdr"
server_key: "274bce032385b8191f6cc6431f9c7162"
server_key_iv: "085bae559e1a299d193e154eb67986cd"
server_key_version: "encryptionkey@0"
I20260619 09:01:54.584069 10978 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781859658134510-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781859658134510-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781859658134510-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 09:01:54.604537 10978 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 09:01:54.606467 10978 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 09:01:54.606586 10978 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 09:01:54.606775 10978 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 09:01:54.607070 10978 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 09:01:54.607102 10978 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 09:01:54.607128 10978 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 09:01:54.607153 10978 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 09:01:54.612294 10978 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33467
I20260619 09:01:54.612352 11106 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33467 every 8 connection(s)
I20260619 09:01:54.612610 10978 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781859658134510-392-0/minicluster-data/ts-2/data/info.pb
I20260619 09:01:54.615216 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ez5NB/build/release/bin/kudu as pid 10978
I20260619 09:01:54.615302 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781859658134510-392-0/minicluster-data/ts-2/wal/instance
I20260619 09:01:54.615461 392 external_mini_cluster.cc:1428] Setting key 0d61e42909af92333546ec6935b65b48
Jun 19 09:01:54 dist-test-slave-mtdr krb5kdc[10603](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859714, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 09:01:54.620877 10706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:54.613235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37093 (local address 127.0.98.62:36633)
0619 09:01:54.613279 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:54.613280 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:54.613462 (+ 182us) server_negotiation.cc:407] Connection header received
0619 09:01:54.613931 (+ 469us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:54.613932 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:54.613943 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:54.613997 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:01:54.614912 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:54.615465 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:54.616616 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:54.616820 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:54.616872 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:54.618741 (+ 1869us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:01:54.618742 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:01:54.618743 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:01:54.618749 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:01:54.619925 (+ 1176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:01:54.620244 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:01:54.620244 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:01:54.620245 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:01:54.620285 (+ 40us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:01:54.620458 (+ 173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:01:54.620458 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:01:54.620459 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:01:54.620543 (+ 84us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:01:54.620586 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0619 09:01:54.620768 (+ 182us) server_negotiation.cc:299] Negotiation successful
0619 09:01:54.620817 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260619 09:01:54.621060 11109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:54.613201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36633 (local address 127.0.98.3:37093)
0619 09:01:54.613349 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0619 09:01:54.613387 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 09:01:54.613831 (+ 444us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:01:54.614083 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:01:54.614087 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:01:54.614604 (+ 517us) client_negotiation.cc:189] Negotiated authn=SASL
0619 09:01:54.614803 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:54.614809 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:54.615545 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:54.615546 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:54.616541 (+ 995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:54.616544 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:54.616886 (+ 342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:54.616887 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:54.617009 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:54.617370 (+ 361us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 09:01:54.617377 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0619 09:01:54.618665 (+ 1288us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 09:01:54.619996 (+ 1331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:01:54.620000 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:01:54.620001 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:01:54.620188 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:01:54.620329 (+ 141us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:01:54.620329 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:01:54.620329 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:01:54.620410 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:01:54.620583 (+ 173us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 09:01:54.620586 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 09:01:54.620685 (+ 99us) client_negotiation.cc:769] Sending connection context
0619 09:01:54.620756 (+ 71us) client_negotiation.cc:240] Negotiation successful
0619 09:01:54.620859 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":66,"threads_started":1}
I20260619 09:01:54.621445 11107 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36633
I20260619 09:01:54.621526 11107 heartbeater.cc:464] Registering TS with master...
I20260619 09:01:54.621680 11107 heartbeater.cc:511] Master 127.0.98.62:36633 requested a full tablet report, sending...
I20260619 09:01:54.621990 10648 ts_manager.cc:194] Registered new tserver with Master: 0d5b0c5586634312bf0ff0f501bfe75c (127.0.98.3:33467)
I20260619 09:01:54.622318 10648 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:37093
I20260619 09:01:54.627050 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 09:01:54.633044 10706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:54.628166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59580 (local address 127.0.98.62:36633)
0619 09:01:54.628201 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:54.628202 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:54.628314 (+ 112us) server_negotiation.cc:407] Connection header received
0619 09:01:54.628349 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:54.628350 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:54.628357 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:54.628401 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:01:54.628885 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:54.629318 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:54.630024 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:54.630190 (+ 166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:54.630230 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:54.631287 (+ 1057us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:01:54.631288 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:01:54.631288 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:01:54.631292 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:01:54.632273 (+ 981us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:01:54.632538 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:01:54.632539 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:01:54.632539 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:01:54.632577 (+ 38us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:01:54.632693 (+ 116us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:01:54.632694 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:01:54.632694 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:01:54.632801 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:01:54.632863 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0619 09:01:54.632939 (+ 76us) server_negotiation.cc:299] Negotiation successful
0619 09:01:54.632988 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260619 09:01:54.634444 10648 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59580:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 09:01:54.635905 10648 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 09:01:54.641602 11115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:54.637811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39833 (local address 127.0.0.1:49154)
0619 09:01:54.638114 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0619 09:01:54.638122 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0619 09:01:54.638169 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:01:54.638321 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:01:54.638324 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:01:54.638326 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 09:01:54.638382 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:54.638384 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:54.639765 (+ 1381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:54.639767 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:54.640605 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:54.640608 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:54.641404 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:54.641404 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:54.641455 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:54.641465 (+ 10us) client_negotiation.cc:769] Sending connection context
0619 09:01:54.641517 (+ 52us) client_negotiation.cc:240] Negotiation successful
0619 09:01:54.641553 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":51,"threads_started":1}
I20260619 09:01:54.641649 11120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:54.637926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49154 (local address 127.0.98.2:39833)
0619 09:01:54.638099 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:54.638103 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:54.638160 (+ 57us) server_negotiation.cc:407] Connection header received
0619 09:01:54.638232 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:54.638234 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:54.638245 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:54.638322 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 09:01:54.638510 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:54.639678 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:54.640672 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:54.641331 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:54.641406 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:54.641459 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0619 09:01:54.641530 (+ 71us) server_negotiation.cc:299] Negotiation successful
0619 09:01:54.641569 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":93,"threads_started":1}
I20260619 09:01:54.641849 11116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:54.637634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41969 (local address 127.0.0.1:36102)
0619 09:01:54.637982 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0619 09:01:54.638017 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0619 09:01:54.638228 (+ 211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:01:54.638493 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:01:54.638494 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:01:54.638497 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 09:01:54.638545 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:54.638547 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:54.639506 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:54.639507 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:54.640794 (+ 1287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:54.640797 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:54.641644 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:54.641645 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:54.641706 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:54.641710 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 09:01:54.641759 (+ 49us) client_negotiation.cc:240] Negotiation successful
0619 09:01:54.641792 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":339,"mutex_wait_us":48,"thread_start_us":45,"threads_started":1}
I20260619 09:01:54.642025 11117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:54.637729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36102 (local address 127.0.98.1:41969)
0619 09:01:54.637912 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:54.637917 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:54.638087 (+ 170us) server_negotiation.cc:407] Connection header received
0619 09:01:54.638318 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:54.638320 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:54.638335 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:54.638418 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 09:01:54.638643 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:54.639420 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:54.640875 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:54.641568 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:54.641631 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:54.641683 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0619 09:01:54.641908 (+ 225us) server_negotiation.cc:299] Negotiation successful
0619 09:01:54.641952 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":76,"threads_started":1}
I20260619 09:01:54.642685 11119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:54.637634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33467 (local address 127.0.0.1:38374)
0619 09:01:54.637968 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0619 09:01:54.638007 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 09:01:54.638075 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:01:54.638349 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:01:54.638350 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:01:54.638352 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 09:01:54.638387 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:54.638388 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:54.639644 (+ 1256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:54.639645 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:54.640795 (+ 1150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:54.640797 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:54.642474 (+ 1677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:54.642475 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:54.642536 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:54.642539 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 09:01:54.642586 (+ 47us) client_negotiation.cc:240] Negotiation successful
0619 09:01:54.642622 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":310,"mutex_wait_us":6,"thread_start_us":50,"threads_started":1}
I20260619 09:01:54.642607 10907 tablet_service.cc:1467] Processing CreateTablet for tablet 0b187a7abb344677a0e5bd967275c097 (DEFAULT_TABLE table=test-table [id=dbf30137cf2348d699019ff5dd8692ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 09:01:54.642655 10773 tablet_service.cc:1467] Processing CreateTablet for tablet 0b187a7abb344677a0e5bd967275c097 (DEFAULT_TABLE table=test-table [id=dbf30137cf2348d699019ff5dd8692ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 09:01:54.642949 10907 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b187a7abb344677a0e5bd967275c097. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 09:01:54.642956 11118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:54.637764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38374 (local address 127.0.98.3:33467)
0619 09:01:54.637913 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:54.637917 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:54.638163 (+ 246us) server_negotiation.cc:407] Connection header received
0619 09:01:54.638201 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:54.638203 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:54.638218 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:54.638284 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 09:01:54.638463 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:54.639551 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:54.641718 (+ 2167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:54.642322 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:54.642383 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:54.642429 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0619 09:01:54.642842 (+ 413us) server_negotiation.cc:299] Negotiation successful
0619 09:01:54.642883 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":70,"threads_started":1}
I20260619 09:01:54.643178 10773 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b187a7abb344677a0e5bd967275c097. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 09:01:54.643486 11041 tablet_service.cc:1467] Processing CreateTablet for tablet 0b187a7abb344677a0e5bd967275c097 (DEFAULT_TABLE table=test-table [id=dbf30137cf2348d699019ff5dd8692ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 09:01:54.643764 11041 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b187a7abb344677a0e5bd967275c097. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 09:01:54.645514 11121 tablet_bootstrap.cc:492] T 0b187a7abb344677a0e5bd967275c097 P ae32eaeaae174261ab3a2d65409f77d9: Bootstrap starting.
I20260619 09:01:54.646199 11121 tablet_bootstrap.cc:654] T 0b187a7abb344677a0e5bd967275c097 P ae32eaeaae174261ab3a2d65409f77d9: Neither blocks nor log segments found. Creating new log.
I20260619 09:01:54.646289 11122 tablet_bootstrap.cc:492] T 0b187a7abb344677a0e5bd967275c097 P f057e8287b944303bb8a54ab05d3f496: Bootstrap starting.
I20260619 09:01:54.646646 11123 tablet_bootstrap.cc:492] T 0b187a7abb344677a0e5bd967275c097 P 0d5b0c5586634312bf0ff0f501bfe75c: Bootstrap starting.
I20260619 09:01:54.646839 11121 log.cc:824] T 0b187a7abb344677a0e5bd967275c097 P ae32eaeaae174261ab3a2d65409f77d9: Log is configured to *not* fsync() on all Append() calls
I20260619 09:01:54.646945 11122 tablet_bootstrap.cc:654] T 0b187a7abb344677a0e5bd967275c097 P f057e8287b944303bb8a54ab05d3f496: Neither blocks nor log segments found. Creating new log.
I20260619 09:01:54.647320 11123 tablet_bootstrap.cc:654] T 0b187a7abb344677a0e5bd967275c097 P 0d5b0c5586634312bf0ff0f501bfe75c: Neither blocks nor log segments found. Creating new log.
I20260619 09:01:54.647399 11122 log.cc:824] T 0b187a7abb344677a0e5bd967275c097 P f057e8287b944303bb8a54ab05d3f496: Log is configured to *not* fsync() on all Append() calls
I20260619 09:01:54.647604 11121 tablet_bootstrap.cc:492] T 0b187a7abb344677a0e5bd967275c097 P ae32eaeaae174261ab3a2d65409f77d9: No bootstrap required, opened a new log
I20260619 09:01:54.647670 11121 ts_tablet_manager.cc:1397] T 0b187a7abb344677a0e5bd967275c097 P ae32eaeaae174261ab3a2d65409f77d9: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260619 09:01:54.647810 11123 log.cc:824] T 0b187a7abb344677a0e5bd967275c097 P 0d5b0c5586634312bf0ff0f501bfe75c: Log is configured to *not* fsync() on all Append() calls
I20260619 09:01:54.648180 11122 tablet_bootstrap.cc:492] T 0b187a7abb344677a0e5bd967275c097 P f057e8287b944303bb8a54ab05d3f496: No bootstrap required, opened a new log
I20260619 09:01:54.648289 11122 ts_tablet_manager.cc:1397] T 0b187a7abb344677a0e5bd967275c097 P f057e8287b944303bb8a54ab05d3f496: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260619 09:01:54.648444 11123 tablet_bootstrap.cc:492] T 0b187a7abb344677a0e5bd967275c097 P 0d5b0c5586634312bf0ff0f501bfe75c: No bootstrap required, opened a new log
I20260619 09:01:54.648499 11123 ts_tablet_manager.cc:1397] T 0b187a7abb344677a0e5bd967275c097 P 0d5b0c5586634312bf0ff0f501bfe75c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 09:01:54.649582 11121 raft_consensus.cc:348] T 0b187a7abb344677a0e5bd967275c097 P ae32eaeaae174261ab3a2d65409f77d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d5b0c5586634312bf0ff0f501bfe75c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33467 } } peers { permanent_uuid: "f057e8287b944303bb8a54ab05d3f496" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41969 } } peers { permanent_uuid: "ae32eaeaae174261ab3a2d65409f77d9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39833 } }
I20260619 09:01:54.649753 11121 raft_consensus.cc:374] T 0b187a7abb344677a0e5bd967275c097 P ae32eaeaae174261ab3a2d65409f77d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 09:01:54.649792 11121 raft_consensus.cc:729] T 0b187a7abb344677a0e5bd967275c097 P ae32eaeaae174261ab3a2d65409f77d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae32eaeaae174261ab3a2d65409f77d9, State: Initialized, Role: FOLLOWER
I20260619 09:01:54.649925 11121 consensus_queue.cc:260] T 0b187a7abb344677a0e5bd967275c097 P ae32eaeaae174261ab3a2d65409f77d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d5b0c5586634312bf0ff0f501bfe75c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33467 } } peers { permanent_uuid: "f057e8287b944303bb8a54ab05d3f496" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41969 } } peers { permanent_uuid: "ae32eaeaae174261ab3a2d65409f77d9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39833 } }
I20260619 09:01:54.650116 11123 raft_consensus.cc:348] T 0b187a7abb344677a0e5bd967275c097 P 0d5b0c5586634312bf0ff0f501bfe75c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d5b0c5586634312bf0ff0f501bfe75c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33467 } } peers { permanent_uuid: "f057e8287b944303bb8a54ab05d3f496" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41969 } } peers { permanent_uuid: "ae32eaeaae174261ab3a2d65409f77d9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39833 } }
I20260619 09:01:54.650249 10973 heartbeater.cc:503] Master 127.0.98.62:36633 was elected leader, sending a full tablet report...
I20260619 09:01:54.650282 11123 raft_consensus.cc:374] T 0b187a7abb344677a0e5bd967275c097 P 0d5b0c5586634312bf0ff0f501bfe75c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 09:01:54.650322 11123 raft_consensus.cc:729] T 0b187a7abb344677a0e5bd967275c097 P 0d5b0c5586634312bf0ff0f501bfe75c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d5b0c5586634312bf0ff0f501bfe75c, State: Initialized, Role: FOLLOWER
I20260619 09:01:54.650197 11121 ts_tablet_manager.cc:1428] T 0b187a7abb344677a0e5bd967275c097 P ae32eaeaae174261ab3a2d65409f77d9: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260619 09:01:54.650429 11123 consensus_queue.cc:260] T 0b187a7abb344677a0e5bd967275c097 P 0d5b0c5586634312bf0ff0f501bfe75c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d5b0c5586634312bf0ff0f501bfe75c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33467 } } peers { permanent_uuid: "f057e8287b944303bb8a54ab05d3f496" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41969 } } peers { permanent_uuid: "ae32eaeaae174261ab3a2d65409f77d9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39833 } }
I20260619 09:01:54.650543 11122 raft_consensus.cc:348] T 0b187a7abb344677a0e5bd967275c097 P f057e8287b944303bb8a54ab05d3f496 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d5b0c5586634312bf0ff0f501bfe75c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33467 } } peers { permanent_uuid: "f057e8287b944303bb8a54ab05d3f496" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41969 } } peers { permanent_uuid: "ae32eaeaae174261ab3a2d65409f77d9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39833 } }
I20260619 09:01:54.650671 11123 ts_tablet_manager.cc:1428] T 0b187a7abb344677a0e5bd967275c097 P 0d5b0c5586634312bf0ff0f501bfe75c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 09:01:54.650693 11122 raft_consensus.cc:374] T 0b187a7abb344677a0e5bd967275c097 P f057e8287b944303bb8a54ab05d3f496 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 09:01:54.650735 11122 raft_consensus.cc:729] T 0b187a7abb344677a0e5bd967275c097 P f057e8287b944303bb8a54ab05d3f496 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f057e8287b944303bb8a54ab05d3f496, State: Initialized, Role: FOLLOWER
I20260619 09:01:54.650766 11107 heartbeater.cc:503] Master 127.0.98.62:36633 was elected leader, sending a full tablet report...
I20260619 09:01:54.650878 11122 consensus_queue.cc:260] T 0b187a7abb344677a0e5bd967275c097 P f057e8287b944303bb8a54ab05d3f496 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d5b0c5586634312bf0ff0f501bfe75c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33467 } } peers { permanent_uuid: "f057e8287b944303bb8a54ab05d3f496" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41969 } } peers { permanent_uuid: "ae32eaeaae174261ab3a2d65409f77d9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39833 } }
I20260619 09:01:54.651108 11122 ts_tablet_manager.cc:1428] T 0b187a7abb344677a0e5bd967275c097 P f057e8287b944303bb8a54ab05d3f496: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260619 09:01:54.651226 10839 heartbeater.cc:503] Master 127.0.98.62:36633 was elected leader, sending a full tablet report...
W20260619 09:01:54.714913 10974 tablet.cc:2369] T 0b187a7abb344677a0e5bd967275c097 P ae32eaeaae174261ab3a2d65409f77d9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 09:01:54.812357 11128 raft_consensus.cc:482] T 0b187a7abb344677a0e5bd967275c097 P 0d5b0c5586634312bf0ff0f501bfe75c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 09:01:54.812479 11128 raft_consensus.cc:504] T 0b187a7abb344677a0e5bd967275c097 P 0d5b0c5586634312bf0ff0f501bfe75c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d5b0c5586634312bf0ff0f501bfe75c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33467 } } peers { permanent_uuid: "f057e8287b944303bb8a54ab05d3f496" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41969 } } peers { permanent_uuid: "ae32eaeaae174261ab3a2d65409f77d9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39833 } }
I20260619 09:01:54.812845 11128 leader_election.cc:288] T 0b187a7abb344677a0e5bd967275c097 P 0d5b0c5586634312bf0ff0f501bfe75c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f057e8287b944303bb8a54ab05d3f496 (127.0.98.1:41969), ae32eaeaae174261ab3a2d65409f77d9 (127.0.98.2:39833)
I20260619 09:01:54.815791 11120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:54.813115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45107 (local address 127.0.98.2:39833)
0619 09:01:54.813153 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:54.813155 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:54.813310 (+ 155us) server_negotiation.cc:407] Connection header received
0619 09:01:54.813345 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:54.813346 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:54.813359 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:54.813412 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 09:01:54.813579 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:54.814062 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:54.815054 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:54.815466 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:54.815534 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:54.815545 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0619 09:01:54.815678 (+ 133us) server_negotiation.cc:299] Negotiation successful
0619 09:01:54.815717 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260619 09:01:54.815804 11117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:54.813074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45209 (local address 127.0.98.1:41969)
0619 09:01:54.813122 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:54.813123 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:54.813165 (+ 42us) server_negotiation.cc:407] Connection header received
0619 09:01:54.813228 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:54.813229 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:54.813244 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:54.813303 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 09:01:54.813457 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:54.813950 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:54.815124 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:54.815472 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:54.815531 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:54.815541 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0619 09:01:54.815707 (+ 166us) server_negotiation.cc:299] Negotiation successful
0619 09:01:54.815746 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260619 09:01:54.815799 11130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:54.813016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39833 (local address 127.0.98.3:45107)
0619 09:01:54.813224 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0619 09:01:54.813234 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 09:01:54.813303 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:01:54.813417 (+ 114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:01:54.813418 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:01:54.813419 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 09:01:54.813484 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:54.813485 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:54.814140 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:54.814141 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:54.814970 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:54.814972 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:54.815548 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:54.815549 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:54.815611 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:54.815615 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 09:01:54.815655 (+ 40us) client_negotiation.cc:240] Negotiation successful
0619 09:01:54.815689 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191,"mutex_wait_us":49,"thread_start_us":88,"threads_started":1}
I20260619 09:01:54.815874 11109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:54.813006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41969 (local address 127.0.98.3:45209)
0619 09:01:54.813109 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0619 09:01:54.813121 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 09:01:54.813174 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:01:54.813322 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:01:54.813323 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:01:54.813325 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 09:01:54.813394 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:54.813397 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:54.814041 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:54.814042 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:54.815034 (+ 992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:01:54.815037 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:01:54.815547 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:01:54.815548 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:01:54.815633 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:54.815636 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 09:01:54.815696 (+ 60us) client_negotiation.cc:240] Negotiation successful
0619 09:01:54.815734 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":84,"mutex_wait_us":1}
I20260619 09:01:54.816318 10927 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b187a7abb344677a0e5bd967275c097" candidate_uuid: "0d5b0c5586634312bf0ff0f501bfe75c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae32eaeaae174261ab3a2d65409f77d9" is_pre_election: true
I20260619 09:01:54.816344 10793 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b187a7abb344677a0e5bd967275c097" candidate_uuid: "0d5b0c5586634312bf0ff0f501bfe75c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f057e8287b944303bb8a54ab05d3f496" is_pre_election: true
I20260619 09:01:54.816488 10927 raft_consensus.cc:2454] T 0b187a7abb344677a0e5bd967275c097 P ae32eaeaae174261ab3a2d65409f77d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d5b0c5586634312bf0ff0f501bfe75c in term 0.
I20260619 09:01:54.816488 10793 raft_consensus.cc:2454] T 0b187a7abb344677a0e5bd967275c097 P f057e8287b944303bb8a54ab05d3f496 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d5b0c5586634312bf0ff0f501bfe75c in term 0.
I20260619 09:01:54.816707 10997 leader_election.cc:302] T 0b187a7abb344677a0e5bd967275c097 P 0d5b0c5586634312bf0ff0f501bfe75c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d5b0c5586634312bf0ff0f501bfe75c, ae32eaeaae174261ab3a2d65409f77d9; no voters:
I20260619 09:01:54.816908 11128 raft_consensus.cc:2781] T 0b187a7abb344677a0e5bd967275c097 P 0d5b0c5586634312bf0ff0f501bfe75c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 09:01:54.816984 11128 raft_consensus.cc:482] T 0b187a7abb344677a0e5bd967275c097 P 0d5b0c5586634312bf0ff0f501bfe75c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 09:01:54.817008 11128 raft_consensus.cc:3037] T 0b187a7abb344677a0e5bd967275c097 P 0d5b0c5586634312bf0ff0f501bfe75c [term 0 FOLLOWER]: Advancing to term 1
I20260619 09:01:54.817744 11128 raft_consensus.cc:504] T 0b187a7abb344677a0e5bd967275c097 P 0d5b0c5586634312bf0ff0f501bfe75c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d5b0c5586634312bf0ff0f501bfe75c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33467 } } peers { permanent_uuid: "f057e8287b944303bb8a54ab05d3f496" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41969 } } peers { permanent_uuid: "ae32eaeaae174261ab3a2d65409f77d9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39833 } }
I20260619 09:01:54.817853 11128 leader_election.cc:288] T 0b187a7abb344677a0e5bd967275c097 P 0d5b0c5586634312bf0ff0f501bfe75c [CANDIDATE]: Term 1 election: Requested vote from peers f057e8287b944303bb8a54ab05d3f496 (127.0.98.1:41969), ae32eaeaae174261ab3a2d65409f77d9 (127.0.98.2:39833)
I20260619 09:01:54.818075 10927 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b187a7abb344677a0e5bd967275c097" candidate_uuid: "0d5b0c5586634312bf0ff0f501bfe75c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae32eaeaae174261ab3a2d65409f77d9"
I20260619 09:01:54.818105 10793 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b187a7abb344677a0e5bd967275c097" candidate_uuid: "0d5b0c5586634312bf0ff0f501bfe75c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f057e8287b944303bb8a54ab05d3f496"
I20260619 09:01:54.818171 10793 raft_consensus.cc:3037] T 0b187a7abb344677a0e5bd967275c097 P f057e8287b944303bb8a54ab05d3f496 [term 0 FOLLOWER]: Advancing to term 1
I20260619 09:01:54.818168 10927 raft_consensus.cc:3037] T 0b187a7abb344677a0e5bd967275c097 P ae32eaeaae174261ab3a2d65409f77d9 [term 0 FOLLOWER]: Advancing to term 1
I20260619 09:01:54.819109 10927 raft_consensus.cc:2454] T 0b187a7abb344677a0e5bd967275c097 P ae32eaeaae174261ab3a2d65409f77d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d5b0c5586634312bf0ff0f501bfe75c in term 1.
I20260619 09:01:54.819109 10793 raft_consensus.cc:2454] T 0b187a7abb344677a0e5bd967275c097 P f057e8287b944303bb8a54ab05d3f496 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d5b0c5586634312bf0ff0f501bfe75c in term 1.
I20260619 09:01:54.819316 10997 leader_election.cc:302] T 0b187a7abb344677a0e5bd967275c097 P 0d5b0c5586634312bf0ff0f501bfe75c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d5b0c5586634312bf0ff0f501bfe75c, ae32eaeaae174261ab3a2d65409f77d9; no voters:
I20260619 09:01:54.819403 11128 raft_consensus.cc:2781] T 0b187a7abb344677a0e5bd967275c097 P 0d5b0c5586634312bf0ff0f501bfe75c [term 1 FOLLOWER]: Leader election won for term 1
I20260619 09:01:54.819561 11128 raft_consensus.cc:686] T 0b187a7abb344677a0e5bd967275c097 P 0d5b0c5586634312bf0ff0f501bfe75c [term 1 LEADER]: Becoming Leader. State: Replica: 0d5b0c5586634312bf0ff0f501bfe75c, State: Running, Role: LEADER
I20260619 09:01:54.819684 11128 consensus_queue.cc:237] T 0b187a7abb344677a0e5bd967275c097 P 0d5b0c5586634312bf0ff0f501bfe75c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d5b0c5586634312bf0ff0f501bfe75c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33467 } } peers { permanent_uuid: "f057e8287b944303bb8a54ab05d3f496" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41969 } } peers { permanent_uuid: "ae32eaeaae174261ab3a2d65409f77d9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39833 } }
I20260619 09:01:54.820508 10647 catalog_manager.cc:5495] T 0b187a7abb344677a0e5bd967275c097 P 0d5b0c5586634312bf0ff0f501bfe75c reported cstate change: term changed from 0 to 1, leader changed from <none> to 0d5b0c5586634312bf0ff0f501bfe75c (127.0.98.3). New cstate: current_term: 1 leader_uuid: "0d5b0c5586634312bf0ff0f501bfe75c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d5b0c5586634312bf0ff0f501bfe75c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33467 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f057e8287b944303bb8a54ab05d3f496" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41969 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae32eaeaae174261ab3a2d65409f77d9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39833 } health_report { overall_health: UNKNOWN } } }
W20260619 09:01:54.825356 10840 tablet.cc:2369] T 0b187a7abb344677a0e5bd967275c097 P f057e8287b944303bb8a54ab05d3f496: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 09:01:54.863080 11108 tablet.cc:2369] T 0b187a7abb344677a0e5bd967275c097 P 0d5b0c5586634312bf0ff0f501bfe75c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 09:01:54.866428 11118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:54.864224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38386 (local address 127.0.98.3:33467)
0619 09:01:54.864295 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0619 09:01:54.864297 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:01:54.864306 (+ 9us) server_negotiation.cc:407] Connection header received
0619 09:01:54.864335 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:01:54.864336 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:01:54.864354 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:01:54.864419 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 09:01:54.864513 (+ 94us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:54.864981 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:54.865709 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:01:54.865893 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:01:54.865939 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:01:54.866075 (+ 136us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 09:01:54.866165 (+ 90us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 09:01:54.866222 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0619 09:01:54.866326 (+ 104us) server_negotiation.cc:299] Negotiation successful
0619 09:01:54.866343 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 09:01:54.868134 10793 raft_consensus.cc:1261] T 0b187a7abb344677a0e5bd967275c097 P f057e8287b944303bb8a54ab05d3f496 [term 1 FOLLOWER]: Refusing update from remote peer 0d5b0c5586634312bf0ff0f501bfe75c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 09:01:54.868153 10927 raft_consensus.cc:1261] T 0b187a7abb344677a0e5bd967275c097 P ae32eaeaae174261ab3a2d65409f77d9 [term 1 FOLLOWER]: Refusing update from remote peer 0d5b0c5586634312bf0ff0f501bfe75c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 09:01:54.868515 11131 consensus_queue.cc:1035] T 0b187a7abb344677a0e5bd967275c097 P 0d5b0c5586634312bf0ff0f501bfe75c [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae32eaeaae174261ab3a2d65409f77d9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 09:01:54.868633 11128 consensus_queue.cc:1035] T 0b187a7abb344677a0e5bd967275c097 P 0d5b0c5586634312bf0ff0f501bfe75c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f057e8287b944303bb8a54ab05d3f496" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 09:01:54.870854 11134 mvcc.cc:205] Tried to move back new op lower bound from 7298497392096579584 to 7298497391901470720. Current Snapshot: MvccSnapshot[applied={T|T < 7298497392096579584}]
I20260619 09:01:54.871238 11135 mvcc.cc:205] Tried to move back new op lower bound from 7298497392096579584 to 7298497391901470720. Current Snapshot: MvccSnapshot[applied={T|T < 7298497392096579584}]
I20260619 09:01:54.873180 10648 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59580:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 09:01:54.873274 10648 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59580:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 09:01:54.873981 10648 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 14322e8638334ab080cbeffd155ea099: Sending DeleteTablet for 3 replicas of tablet 0b187a7abb344677a0e5bd967275c097
I20260619 09:01:54.874276 11041 tablet_service.cc:1514] Processing DeleteTablet for tablet 0b187a7abb344677a0e5bd967275c097 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 09:01:54 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38374
I20260619 09:01:54.874326 10907 tablet_service.cc:1514] Processing DeleteTablet for tablet 0b187a7abb344677a0e5bd967275c097 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 09:01:54 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49154
I20260619 09:01:54.874331 10773 tablet_service.cc:1514] Processing DeleteTablet for tablet 0b187a7abb344677a0e5bd967275c097 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 09:01:54 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36102
I20260619 09:01:54.874511 11145 tablet_replica.cc:330] T 0b187a7abb344677a0e5bd967275c097 P ae32eaeaae174261ab3a2d65409f77d9: stopping tablet replica
I20260619 09:01:54.874600 11145 raft_consensus.cc:2229] T 0b187a7abb344677a0e5bd967275c097 P ae32eaeaae174261ab3a2d65409f77d9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 09:01:54.874660 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ez5NB/build/release/bin/kudu with pid 10710
I20260619 09:01:54.874698 11145 raft_consensus.cc:2258] T 0b187a7abb344677a0e5bd967275c097 P ae32eaeaae174261ab3a2d65409f77d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 09:01:54.874854 11144 tablet_replica.cc:330] T 0b187a7abb344677a0e5bd967275c097 P 0d5b0c5586634312bf0ff0f501bfe75c: stopping tablet replica
I20260619 09:01:54.874939 11144 raft_consensus.cc:2229] T 0b187a7abb344677a0e5bd967275c097 P 0d5b0c5586634312bf0ff0f501bfe75c [term 1 LEADER]: Raft consensus shutting down.
I20260619 09:01:54.875015 11144 raft_consensus.cc:2258] T 0b187a7abb344677a0e5bd967275c097 P 0d5b0c5586634312bf0ff0f501bfe75c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 09:01:54.875414 11144 ts_tablet_manager.cc:1905] T 0b187a7abb344677a0e5bd967275c097 P 0d5b0c5586634312bf0ff0f501bfe75c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 09:01:54.876976 11144 ts_tablet_manager.cc:1918] T 0b187a7abb344677a0e5bd967275c097 P 0d5b0c5586634312bf0ff0f501bfe75c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 09:01:54.877035 11144 log.cc:1196] T 0b187a7abb344677a0e5bd967275c097 P 0d5b0c5586634312bf0ff0f501bfe75c: Deleting WAL directory at /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781859658134510-392-0/minicluster-data/ts-2/wal/wals/0b187a7abb344677a0e5bd967275c097
I20260619 09:01:54.877288 11144 ts_tablet_manager.cc:1939] T 0b187a7abb344677a0e5bd967275c097 P 0d5b0c5586634312bf0ff0f501bfe75c: Deleting consensus metadata
I20260619 09:01:54.877658 10635 catalog_manager.cc:4841] TS 0d5b0c5586634312bf0ff0f501bfe75c (127.0.98.3:33467): tablet 0b187a7abb344677a0e5bd967275c097 (table test-table [id=dbf30137cf2348d699019ff5dd8692ca]) successfully deleted
I20260619 09:01:54.877792 11145 ts_tablet_manager.cc:1905] T 0b187a7abb344677a0e5bd967275c097 P ae32eaeaae174261ab3a2d65409f77d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 09:01:54.879369 11145 ts_tablet_manager.cc:1918] T 0b187a7abb344677a0e5bd967275c097 P ae32eaeaae174261ab3a2d65409f77d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 09:01:54.879443 11145 log.cc:1196] T 0b187a7abb344677a0e5bd967275c097 P ae32eaeaae174261ab3a2d65409f77d9: Deleting WAL directory at /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781859658134510-392-0/minicluster-data/ts-1/wal/wals/0b187a7abb344677a0e5bd967275c097
I20260619 09:01:54.879709 11145 ts_tablet_manager.cc:1939] T 0b187a7abb344677a0e5bd967275c097 P ae32eaeaae174261ab3a2d65409f77d9: Deleting consensus metadata
I20260619 09:01:54.880069 10635 catalog_manager.cc:4841] TS ae32eaeaae174261ab3a2d65409f77d9 (127.0.98.2:39833): tablet 0b187a7abb344677a0e5bd967275c097 (table test-table [id=dbf30137cf2348d699019ff5dd8692ca]) successfully deleted
W20260619 09:01:54.885691 10632 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:41969 (error 108)
I20260619 09:01:54.885839 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ez5NB/build/release/bin/kudu with pid 10844
I20260619 09:01:54.886765 11119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:01:54.886646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41969 (local address 127.0.0.1:36104)
0619 09:01:54.886687 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0619 09:01:54.886732 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
W20260619 09:01:54.889763 10632 catalog_manager.cc:4584] TS f057e8287b944303bb8a54ab05d3f496 (127.0.98.1:41969): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0b187a7abb344677a0e5bd967275c097: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41969: connect: Connection refused (error 111)
I20260619 09:01:54.889794 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ez5NB/build/release/bin/kudu with pid 10978
I20260619 09:01:54.893522 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ez5NB/build/release/bin/kudu with pid 10616
2026-06-19T09:01:54Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (1973 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859658134510-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 09:01:54 dist-test-slave-mtdr krb5kdc[11150](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 09:01:54 dist-test-slave-mtdr krb5kdc[11150](info): set up 2 sockets
Jun 19 09:01:54 dist-test-slave-mtdr krb5kdc[11150](info): commencing operation
krb5kdc: starting...
W20260619 09:01:55.851428 392 mini_kdc.cc:121] Time spent starting KDC: real 0.940s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859658134510-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859658134510-392-0/krb5kdc/test-user.keytab.
Jun 19 09:01:55 dist-test-slave-mtdr krb5kdc[11150](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859715, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T09:01:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T09:01:55Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859658134510-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859658134510-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859658134510-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859658134510-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859658134510-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859658134510-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
I20260619 09:01:55.995311 392 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859658134510-392-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task4ez5NB/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859658134510-392-0/minicluster-data/postgres -l logfile start
2026-06-19 09:01:58.319 UTC [11180] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-19 09:01:58.319 UTC [11180] LOG: listening on IPv4 address "127.0.98.20", port 36007
2026-06-19 09:01:58.322 UTC [11180] LOG: listening on Unix socket "/tmp/.s.PGSQL.36007"
2026-06-19 09:01:58.342 UTC [11183] LOG: database system was shut down at 2026-06-19 09:01:56 UTC
2026-06-19 09:01:58.346 UTC [11180] LOG: database system is ready to accept connections
I20260619 09:01:59.207259 392 mini_postgres.cc:96] Postgres bound to 36007
2026-06-19 09:01:59.212 UTC [11191] FATAL: database "slave" does not exist
127.0.98.20:36007 - accepting connections
I20260619 09:01:59.213172 392 mini_ranger.cc:161] Starting Ranger...
I20260619 09:01:59.226450 392 mini_ranger.cc:84] Created miniranger Postgres user
I20260619 09:01:59.666604 392 mini_ranger.cc:87] Created ranger Postgres database
I20260619 09:01:59.666692 392 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859658134510-392-0/minicluster-data/ranger-admin
2026-06-19 09:01:59,939 [I] DB FLAVOR :POSTGRES
2026-06-19 09:01:59,940 [I] --------- Verifying Ranger DB connection ---------
2026-06-19 09:01:59,940 [I] Checking connection..
2026-06-19 09:01:59,940 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4ez5NB/build/release/bin/postgresql.jar:/tmp/dist-test-task4ez5NB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:36007/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-19 09:02:00,177 [I] Checking connection passed.
2026-06-19 09:02:00,177 [I] --------- Verifying version history table ---------
2026-06-19 09:02:00,178 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4ez5NB/build/release/bin/postgresql.jar:/tmp/dist-test-task4ez5NB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:36007/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 09:02:00,420 [I] Table x_db_version_h does not exist in database ranger
2026-06-19 09:02:00,420 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4ez5NB/build/release/bin/postgresql.jar:/tmp/dist-test-task4ez5NB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:36007/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 09:02:00,663 [I] Table x_db_version_h does not exist in database ranger
2026-06-19 09:02:00,663 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-19 09:02:00,663 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4ez5NB/build/release/bin/postgresql.jar:/tmp/dist-test-task4ez5NB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:36007/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task4ez5NB/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-19 09:02:00.895 UTC [11255] WARNING: there is no transaction in progress
2026-06-19 09:02:00,900 [I] create_dbversion_catalog.sql file imported successfully
2026-06-19 09:02:00,900 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4ez5NB/build/release/bin/postgresql.jar:/tmp/dist-test-task4ez5NB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:36007/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 09:02:01,149 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-19 09:02:01,149 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-19 09:02:01,149 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4ez5NB/build/release/bin/postgresql.jar:/tmp/dist-test-task4ez5NB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:36007/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-19 09:02:01,398 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4ez5NB/build/release/bin/postgresql.jar:/tmp/dist-test-task4ez5NB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:36007/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-19 09:02:01,624 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4ez5NB/build/release/bin/postgresql.jar:/tmp/dist-test-task4ez5NB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:36007/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-mtdr.c.gcp-upstream.internal','N') ;"
2026-06-19 09:02:01,849 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4ez5NB/build/release/bin/postgresql.jar:/tmp/dist-test-task4ez5NB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:36007/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-19 09:02:02,106 [I] Table x_portal_user does not exist in database ranger
2026-06-19 09:02:02,106 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4ez5NB/build/release/bin/postgresql.jar:/tmp/dist-test-task4ez5NB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:36007/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-19 09:02:02,345 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-19 09:02:02,345 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4ez5NB/build/release/bin/postgresql.jar:/tmp/dist-test-task4ez5NB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:36007/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-19 09:02:02,579 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4ez5NB/build/release/bin/postgresql.jar:/tmp/dist-test-task4ez5NB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:36007/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-19 09:02:02,815 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-19 09:02:02,815 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4ez5NB/build/release/bin/postgresql.jar:/tmp/dist-test-task4ez5NB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:36007/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task4ez5NB/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-19 09:02:03.708 UTC [11363] WARNING: there is no transaction in progress
2026-06-19 09:02:03.724 UTC [11363] WARNING: there is no transaction in progress
2026-06-19 09:02:03.738 UTC [11363] WARNING: there is no transaction in progress
2026-06-19 09:02:03.753 UTC [11363] WARNING: there is no transaction in progress
2026-06-19 09:02:03.769 UTC [11363] WARNING: there is no transaction in progress
2026-06-19 09:02:03.785 UTC [11363] WARNING: there is no transaction in progress
2026-06-19 09:02:03.839 UTC [11363] WARNING: there is no transaction in progress
2026-06-19 09:02:03.846 UTC [11363] WARNING: there is no transaction in progress
2026-06-19 09:02:03.860 UTC [11363] WARNING: there is no transaction in progress
2026-06-19 09:02:03.871 UTC [11363] WARNING: there is no transaction in progress
2026-06-19 09:02:03.880 UTC [11363] WARNING: there is no transaction in progress
2026-06-19 09:02:03.894 UTC [11363] WARNING: there is no transaction in progress
2026-06-19 09:02:03.903 UTC [11363] WARNING: there is no transaction in progress
2026-06-19 09:02:03.909 UTC [11363] WARNING: there is no transaction in progress
2026-06-19 09:02:03.915 UTC [11363] WARNING: there is no transaction in progress
2026-06-19 09:02:03.921 UTC [11363] WARNING: there is no transaction in progress
2026-06-19 09:02:04,515 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-19 09:02:04,515 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4ez5NB/build/release/bin/postgresql.jar:/tmp/dist-test-task4ez5NB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:36007/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-19 09:02:04,757 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-19 09:02:04,757 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4ez5NB/build/release/bin/postgresql.jar:/tmp/dist-test-task4ez5NB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:36007/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-19 09:02:04,980 [I] Table x_portal_user already exists in database 'ranger'
2026-06-19 09:02:04,981 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4ez5NB/build/release/bin/postgresql.jar:/tmp/dist-test-task4ez5NB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:36007/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-19 09:02:05,202 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-19 09:02:05,202 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4ez5NB/build/release/bin/postgresql.jar:/tmp/dist-test-task4ez5NB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:36007/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-19 09:02:05,456 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4ez5NB/build/release/bin/postgresql.jar:/tmp/dist-test-task4ez5NB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:36007/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-19 09:02:05,703 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4ez5NB/build/release/bin/postgresql.jar:/tmp/dist-test-task4ez5NB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:36007/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-mtdr.c.gcp-upstream.internal';"
2026-06-19 09:02:05,936 [I] CORE_DB_SCHEMA import status has been updated
2026-06-19 09:02:05,936 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4ez5NB/build/release/bin/postgresql.jar:/tmp/dist-test-task4ez5NB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:36007/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-19 09:02:06,152 [I] DB_PATCHES have already been applied
I20260619 09:02:06.157998 392 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859658134510-392-0/minicluster-data/ranger-admin:/tmp/dist-test-task4ez5NB/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task4ez5NB/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task4ez5NB/build/release/bin/postgresql.jar:/tmp/dist-test-task4ez5NB/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260619 09:02:06.158064 392 mini_ranger.cc:193] Using host: 127.0.98.20
I20260619 09:02:06.160993 392 mini_ranger.cc:227] Ranger admin URL: http://127.0.98.20:49219
Jun 19, 2026 9:02:06 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 19, 2026 9:02:06 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 19, 2026 9:02:06 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859658134510-392-0/minicluster-data/ranger-admin/ews/webapp
Jun 19, 2026 9:02:06 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859658134510-392-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 19, 2026 9:02:06 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859658134510-392-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 19, 2026 9:02:07 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859658134510-392-0/minicluster-data/ranger-admin/ews/webapp].
Jun 19, 2026 9:02:07 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260619 09:02:07.168087 392 mini_ranger.cc:160] Time spent starting Ranger: real 7.955s user 0.002s sys 0.006s
Jun 19, 2026 9:02:25 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 19, 2026 9:02:25 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.XKeyREST
Jun 19, 2026 9:02:25 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 19, 2026 9:02:25 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 19, 2026 9:02:25 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 19, 2026 9:02:25 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 19, 2026 9:02:25 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 19, 2026 9:02:25 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 19, 2026 9:02:25 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 19, 2026 9:02:25 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 19, 2026 9:02:25 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 19, 2026 9:02:25 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 19, 2026 9:02:25 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 19, 2026 9:02:25 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 19, 2026 9:02:25 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 19, 2026 9:02:25 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 19, 2026 9:02:25 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 19, 2026 9:02:26 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260619 09:02:27.265440 392 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859658134510-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859658134510-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.98.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859658134510-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859658134510-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 19 09:02:27 dist-test-slave-mtdr krb5kdc[11150](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859747, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260619 09:02:27.338393 392 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260619 09:02:27.346352 392 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260619 09:02:28.186882 392 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260619 09:02:28.186954 392 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859658134510-392-0/minicluster-data/ranger-kms
I20260619 09:02:28.186964 392 mini_ranger_kms.cc:227] Using postgres at 127.0.98.20:36007
2026-06-19 09:02:28,345 [I] DB FLAVOR :POSTGRES
2026-06-19 09:02:28,345 [I] --------- Verifying Ranger DB connection ---------
2026-06-19 09:02:28,345 [I] Checking connection
2026-06-19 09:02:28,345 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4ez5NB/build/release/bin/postgresql.jar:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859658134510-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:36007/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-19 09:02:28,597 [I] connection success
2026-06-19 09:02:28,597 [I] --------- Verifying Ranger DB tables ---------
2026-06-19 09:02:28,597 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-19 09:02:28,598 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4ez5NB/build/release/bin/postgresql.jar:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859658134510-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:36007/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-19 09:02:28,838 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-19 09:02:28,838 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-19 09:02:28,839 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-19 09:02:28,839 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4ez5NB/build/release/bin/postgresql.jar:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859658134510-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:36007/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859658134510-392-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-19 09:02:29,090 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260619 09:02:29.522090 392 mini_ranger_kms.cc:310] Created kms service
I20260619 09:02:29.705467 392 mini_ranger_kms.cc:326] Created kudu user
I20260619 09:02:29.746899 392 mini_ranger_kms.cc:343] Created rangerkms user
Jun 19 09:02:29 dist-test-slave-mtdr krb5kdc[11150](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859747, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260619 09:02:30.105032 392 mini_ranger_kms.cc:384] Added ranger policy
I20260619 09:02:30.105301 392 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859658134510-392-0/minicluster-data/ranger-kms:/tmp/dist-test-task4ez5NB/build/release/bin/postgresql.jar:/tmp/dist-test-task4ez5NB/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task4ez5NB/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260619 09:02:30.105325 392 mini_ranger_kms.cc:240] Using host: 127.0.98.20
I20260619 09:02:30.108371 392 mini_ranger_kms.cc:276] Ranger KMS PID: 11543
I20260619 09:02:30.108489 392 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.98.20:36989
Jun 19, 2026 9:02:30 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 19, 2026 9:02:30 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 19, 2026 9:02:31 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859658134510-392-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 19, 2026 9:02:31 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859658134510-392-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 19, 2026 9:02:31 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859658134510-392-0/minicluster-data/ranger-kms/ews/webapp].
Jun 19, 2026 9:02:31 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260619 09:02:32.092386 392 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 4.754s user 0.001s sys 0.139s
I20260619 09:02:32.092476 392 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260619 09:02:32.092551 392 mini_ranger_kms.cc:401] 127.0.98.20:36989/kms/v1/keys
Jun 19 09:02:32 dist-test-slave-mtdr krb5kdc[11150](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781859752, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Jun 19 09:02:32 dist-test-slave-mtdr krb5kdc[11150](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859752, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Jun 19 09:02:32 dist-test-slave-mtdr krb5kdc[11150](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859752, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Jun 19 09:02:35 dist-test-slave-mtdr krb5kdc[11150](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859755, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859658134510-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859658134510-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 09:02:35.161851 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ez5NB/build/release/bin/kudu
/tmp/dist-test-task4ez5NB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859658134510-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859658134510-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859658134510-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859658134510-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33121
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859658134510-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34093
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33121
--ranger_config_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859658134510-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:36989/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859658134510-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859658134510-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859658134510-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 09:02:35.242003 11594 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 09:02:35.242206 11594 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 09:02:35.242234 11594 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 09:02:35.243409 11594 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 09:02:35.243449 11594 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 09:02:35.243468 11594 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 09:02:35.243484 11594 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 09:02:35.243498 11594 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 09:02:35.245023 11594 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34093
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859658134510-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859658134510-392-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.98.20:36989/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33121
--ranger_config_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859658134510-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859658134510-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33121
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859658134510-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11594
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859658134510-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 09:02:35.245254 11594 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 09:02:35.245452 11594 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 09:02:35.250689 11600 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:02:35.250805 11602 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:02:35.250723 11594 server_base.cc:1034] running on GCE node
W20260619 09:02:35.250684 11599 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:02:35.251174 11594 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 09:02:35.251503 11594 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 09:02:35.252674 11594 hybrid_clock.cc:648] HybridClock initialized: now 1781859755252656 us; error 32 us; skew 500 ppm
Jun 19 09:02:35 dist-test-slave-mtdr krb5kdc[11150](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859755, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 09:02:35.255587 11594 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 09:02:35.256093 11594 webserver.cc:460] Webserver started at http://127.0.98.62:33373/ using document root <none> and password file <none>
I20260619 09:02:35.256332 11594 fs_manager.cc:359] Metadata directory not provided
I20260619 09:02:35.256381 11594 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 09:02:35.256503 11594 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 09:02:35 dist-test-slave-mtdr krb5kdc[11150](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859755, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260619 09:02:35.307400 11594 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859658134510-392-0/minicluster-data/master-0/data/instance:
uuid: "63e23856a1e04f25aacd64df8898860d"
format_stamp: "Formatted at 2026-06-19 09:02:35 on dist-test-slave-mtdr"
server_key: "074f6f0924c87149acebf93be06a046c"
server_key_iv: "54a30a88cf3c536e782e842d933c61b7"
server_key_version: "kuduclusterkey@0"
I20260619 09:02:35.307850 11594 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859658134510-392-0/minicluster-data/master-0/wal/instance:
uuid: "63e23856a1e04f25aacd64df8898860d"
format_stamp: "Formatted at 2026-06-19 09:02:35 on dist-test-slave-mtdr"
server_key: "074f6f0924c87149acebf93be06a046c"
server_key_iv: "54a30a88cf3c536e782e842d933c61b7"
server_key_version: "kuduclusterkey@0"
I20260619 09:02:35.309718 11594 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260619 09:02:35.324779 11609 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 09:02:35.325001 11594 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 09:02:35.325094 11594 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859658134510-392-0/minicluster-data/master-0/data,/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859658134510-392-0/minicluster-data/master-0/wal
uuid: "63e23856a1e04f25aacd64df8898860d"
format_stamp: "Formatted at 2026-06-19 09:02:35 on dist-test-slave-mtdr"
server_key: "074f6f0924c87149acebf93be06a046c"
server_key_iv: "54a30a88cf3c536e782e842d933c61b7"
server_key_version: "kuduclusterkey@0"
I20260619 09:02:35.325194 11594 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859658134510-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859658134510-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859658134510-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 09:02:35.337623 11594 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 09:02:35.340265 11594 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 09:02:35.340435 11594 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 09:02:35.345419 11594 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33121
I20260619 09:02:35.345438 11661 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33121 every 8 connection(s)
I20260619 09:02:35.345882 11594 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859658134510-392-0/minicluster-data/master-0/data/info.pb
I20260619 09:02:35.346553 11662 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 09:02:35.347294 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ez5NB/build/release/bin/kudu as pid 11594
I20260619 09:02:35.347388 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859658134510-392-0/minicluster-data/master-0/wal/instance
I20260619 09:02:35.347590 392 external_mini_cluster.cc:1428] Setting key 2d6545230ee25b6386c1d311ca402e46
I20260619 09:02:35.349833 11662 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 63e23856a1e04f25aacd64df8898860d: Bootstrap starting.
I20260619 09:02:35.350389 11662 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 63e23856a1e04f25aacd64df8898860d: Neither blocks nor log segments found. Creating new log.
I20260619 09:02:35.350646 11662 log.cc:824] T 00000000000000000000000000000000 P 63e23856a1e04f25aacd64df8898860d: Log is configured to *not* fsync() on all Append() calls
I20260619 09:02:35.351269 11662 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 63e23856a1e04f25aacd64df8898860d: No bootstrap required, opened a new log
I20260619 09:02:35.353327 11662 raft_consensus.cc:348] T 00000000000000000000000000000000 P 63e23856a1e04f25aacd64df8898860d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63e23856a1e04f25aacd64df8898860d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33121 } }
I20260619 09:02:35.353513 11662 raft_consensus.cc:374] T 00000000000000000000000000000000 P 63e23856a1e04f25aacd64df8898860d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 09:02:35.353601 11662 raft_consensus.cc:729] T 00000000000000000000000000000000 P 63e23856a1e04f25aacd64df8898860d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63e23856a1e04f25aacd64df8898860d, State: Initialized, Role: FOLLOWER
I20260619 09:02:35.353765 11662 consensus_queue.cc:260] T 00000000000000000000000000000000 P 63e23856a1e04f25aacd64df8898860d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63e23856a1e04f25aacd64df8898860d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33121 } }
I20260619 09:02:35.353857 11662 raft_consensus.cc:388] T 00000000000000000000000000000000 P 63e23856a1e04f25aacd64df8898860d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 09:02:35.353890 11662 raft_consensus.cc:482] T 00000000000000000000000000000000 P 63e23856a1e04f25aacd64df8898860d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 09:02:35.353951 11662 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 63e23856a1e04f25aacd64df8898860d [term 0 FOLLOWER]: Advancing to term 1
I20260619 09:02:35.354616 11662 raft_consensus.cc:504] T 00000000000000000000000000000000 P 63e23856a1e04f25aacd64df8898860d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63e23856a1e04f25aacd64df8898860d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33121 } }
I20260619 09:02:35.354725 11662 leader_election.cc:302] T 00000000000000000000000000000000 P 63e23856a1e04f25aacd64df8898860d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 63e23856a1e04f25aacd64df8898860d; no voters:
I20260619 09:02:35.354954 11662 leader_election.cc:288] T 00000000000000000000000000000000 P 63e23856a1e04f25aacd64df8898860d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 09:02:35.355002 11667 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 63e23856a1e04f25aacd64df8898860d [term 1 FOLLOWER]: Leader election won for term 1
I20260619 09:02:35.355245 11667 raft_consensus.cc:686] T 00000000000000000000000000000000 P 63e23856a1e04f25aacd64df8898860d [term 1 LEADER]: Becoming Leader. State: Replica: 63e23856a1e04f25aacd64df8898860d, State: Running, Role: LEADER
I20260619 09:02:35.355312 11662 sys_catalog.cc:564] T 00000000000000000000000000000000 P 63e23856a1e04f25aacd64df8898860d [sys.catalog]: configured and running, proceeding with master startup.
I20260619 09:02:35.355383 11667 consensus_queue.cc:237] T 00000000000000000000000000000000 P 63e23856a1e04f25aacd64df8898860d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63e23856a1e04f25aacd64df8898860d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33121 } }
Jun 19 09:02:35 dist-test-slave-mtdr krb5kdc[11150](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859755, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 09:02:35.355787 11667 sys_catalog.cc:455] T 00000000000000000000000000000000 P 63e23856a1e04f25aacd64df8898860d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "63e23856a1e04f25aacd64df8898860d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63e23856a1e04f25aacd64df8898860d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33121 } } }
I20260619 09:02:35.355929 11667 sys_catalog.cc:458] T 00000000000000000000000000000000 P 63e23856a1e04f25aacd64df8898860d [sys.catalog]: This master's current role is: LEADER
I20260619 09:02:35.356093 11667 sys_catalog.cc:455] T 00000000000000000000000000000000 P 63e23856a1e04f25aacd64df8898860d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 63e23856a1e04f25aacd64df8898860d. Latest consensus state: current_term: 1 leader_uuid: "63e23856a1e04f25aacd64df8898860d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63e23856a1e04f25aacd64df8898860d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33121 } } }
I20260619 09:02:35.356190 11667 sys_catalog.cc:458] T 00000000000000000000000000000000 P 63e23856a1e04f25aacd64df8898860d [sys.catalog]: This master's current role is: LEADER
I20260619 09:02:35.356328 11676 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 09:02:35.357136 11676 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 09:02:35.359081 11676 catalog_manager.cc:1347] Generated new cluster ID: 8f3569c617a342edbfbb6db8b62e3371
I20260619 09:02:35.359125 11676 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 09:02:35.360142 11665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:02:35.348176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50394 (local address 127.0.98.62:33121)
0619 09:02:35.348427 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0619 09:02:35.348437 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 09:02:35.348468 (+ 31us) server_negotiation.cc:407] Connection header received
0619 09:02:35.348989 (+ 521us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:02:35.348998 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:02:35.349059 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:02:35.349215 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:02:35.349996 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:02:35.351536 (+ 1540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:02:35.352591 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:02:35.353252 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:02:35.353373 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:02:35.356330 (+ 2957us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:02:35.356335 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:02:35.356335 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:02:35.356348 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:02:35.358164 (+ 1816us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:02:35.358549 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:02:35.358553 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:02:35.358553 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:02:35.358642 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:02:35.358856 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:02:35.358857 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:02:35.358857 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:02:35.359142 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:02:35.359292 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0619 09:02:35.359822 (+ 530us) server_negotiation.cc:299] Negotiation successful
0619 09:02:35.359941 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":203,"thread_start_us":138,"threads_started":1}
I20260619 09:02:35.360357 11662 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859658134510-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260619 09:02:35.374078 11676 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 09:02:35.375175 11676 catalog_manager.cc:1504] Loading token signing keys...
I20260619 09:02:35.383167 11676 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 63e23856a1e04f25aacd64df8898860d: Generated new TSK 0
I20260619 09:02:35.383430 11676 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859658134510-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859658134510-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 09:02:36.156965 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ez5NB/build/release/bin/kudu
/tmp/dist-test-task4ez5NB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859658134510-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859658134510-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859658134510-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859658134510-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:36989/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33121
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859658134510-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34093
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859658134510-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859658134510-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859658134510-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 09:02:36.266039 11707 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 09:02:36.266376 11707 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 09:02:36.266449 11707 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 09:02:36.268208 11707 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 09:02:36.268317 11707 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 09:02:36.268394 11707 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 09:02:36.270505 11707 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34093
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859658134510-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859658134510-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.98.20:36989/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859658134510-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859658134510-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33121
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11707
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859658134510-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 09:02:36.270961 11707 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 09:02:36.271184 11707 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 09:02:36.275802 11713 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:02:36.276060 11712 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:02:36.276203 11715 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:02:36.279018 11707 server_base.cc:1034] running on GCE node
I20260619 09:02:36.279220 11707 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 09:02:36.279467 11707 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 09:02:36.280642 11707 hybrid_clock.cc:648] HybridClock initialized: now 1781859756280617 us; error 34 us; skew 500 ppm
Jun 19 09:02:36 dist-test-slave-mtdr krb5kdc[11150](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859756, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 09:02:36.283079 11707 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 09:02:36.283473 11707 webserver.cc:460] Webserver started at http://127.0.98.1:40307/ using document root <none> and password file <none>
I20260619 09:02:36.283669 11707 fs_manager.cc:359] Metadata directory not provided
I20260619 09:02:36.283718 11707 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 09:02:36.283840 11707 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 09:02:36 dist-test-slave-mtdr krb5kdc[11150](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859756, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260619 09:02:36.296473 11707 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859658134510-392-0/minicluster-data/ts-0/data/instance:
uuid: "cd3b555594c449ceb5e0ce4a7f38d4f8"
format_stamp: "Formatted at 2026-06-19 09:02:36 on dist-test-slave-mtdr"
server_key: "c172c669a6fe2f29402394531cfe904e"
server_key_iv: "a110fa92f42e20222fa689b6a8088d05"
server_key_version: "kuduclusterkey@0"
I20260619 09:02:36.296944 11707 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859658134510-392-0/minicluster-data/ts-0/wal/instance:
uuid: "cd3b555594c449ceb5e0ce4a7f38d4f8"
format_stamp: "Formatted at 2026-06-19 09:02:36 on dist-test-slave-mtdr"
server_key: "c172c669a6fe2f29402394531cfe904e"
server_key_iv: "a110fa92f42e20222fa689b6a8088d05"
server_key_version: "kuduclusterkey@0"
I20260619 09:02:36.299757 11707 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.002s
I20260619 09:02:36.313171 11722 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 09:02:36.313397 11707 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 09:02:36.313477 11707 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859658134510-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859658134510-392-0/minicluster-data/ts-0/wal
uuid: "cd3b555594c449ceb5e0ce4a7f38d4f8"
format_stamp: "Formatted at 2026-06-19 09:02:36 on dist-test-slave-mtdr"
server_key: "c172c669a6fe2f29402394531cfe904e"
server_key_iv: "a110fa92f42e20222fa689b6a8088d05"
server_key_version: "kuduclusterkey@0"
I20260619 09:02:36.313561 11707 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859658134510-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859658134510-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859658134510-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 09:02:36.327172 11707 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 09:02:36.328915 11707 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 09:02:36.329044 11707 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 09:02:36.329367 11707 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 09:02:36.329702 11707 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 09:02:36.329738 11707 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 09:02:36.329766 11707 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 09:02:36.329802 11707 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 09:02:36.337196 11707 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39133
I20260619 09:02:36.337576 11707 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859658134510-392-0/minicluster-data/ts-0/data/info.pb
I20260619 09:02:36.338147 11835 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39133 every 8 connection(s)
Jun 19 09:02:36 dist-test-slave-mtdr krb5kdc[11150](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859756, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 09:02:36.346959 11839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:02:36.338402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33121 (local address 127.0.98.1:42139)
0619 09:02:36.338612 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0619 09:02:36.338651 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 09:02:36.339177 (+ 526us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:02:36.339452 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:02:36.339458 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:02:36.339633 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0619 09:02:36.339829 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:02:36.339833 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:02:36.340514 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:02:36.340515 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:02:36.341311 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:02:36.341313 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:02:36.341668 (+ 355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:02:36.341669 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:02:36.341800 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:02:36.342292 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 09:02:36.342300 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 09:02:36.343532 (+ 1232us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 09:02:36.345399 (+ 1867us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:02:36.345405 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:02:36.345407 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:02:36.345671 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:02:36.345887 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:02:36.345887 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:02:36.345888 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:02:36.346006 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:02:36.346285 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 09:02:36.346290 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 09:02:36.346455 (+ 165us) client_negotiation.cc:769] Sending connection context
0619 09:02:36.346553 (+ 98us) client_negotiation.cc:240] Negotiation successful
0619 09:02:36.346698 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":83,"threads_started":1}
I20260619 09:02:36.347551 11838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:02:36.338406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42139 (local address 127.0.98.62:33121)
0619 09:02:36.338557 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0619 09:02:36.338559 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:02:36.338743 (+ 184us) server_negotiation.cc:407] Connection header received
0619 09:02:36.339302 (+ 559us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:02:36.339303 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:02:36.339317 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:02:36.339378 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:02:36.339918 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:02:36.340432 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:02:36.341397 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:02:36.341604 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:02:36.341658 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:02:36.343617 (+ 1959us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:02:36.343619 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:02:36.343619 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:02:36.343625 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:02:36.345235 (+ 1610us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:02:36.345761 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:02:36.345763 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:02:36.345763 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:02:36.345825 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:02:36.346069 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:02:36.346070 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:02:36.346071 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:02:36.346203 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:02:36.346260 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0619 09:02:36.347374 (+ 1114us) server_negotiation.cc:299] Negotiation successful
0619 09:02:36.347456 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":84,"threads_started":1}
I20260619 09:02:36.347957 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ez5NB/build/release/bin/kudu as pid 11707
I20260619 09:02:36.348042 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859658134510-392-0/minicluster-data/ts-0/wal/instance
I20260619 09:02:36.348212 392 external_mini_cluster.cc:1428] Setting key eb58ec438cd405036a09be7936d4ba64
I20260619 09:02:36.348913 11836 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33121
I20260619 09:02:36.349026 11836 heartbeater.cc:464] Registering TS with master...
I20260619 09:02:36.349226 11836 heartbeater.cc:511] Master 127.0.98.62:33121 requested a full tablet report, sending...
I20260619 09:02:36.350653 11626 ts_manager.cc:194] Registered new tserver with Master: cd3b555594c449ceb5e0ce4a7f38d4f8 (127.0.98.1:39133)
I20260619 09:02:36.351534 11626 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:42139
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859658134510-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859658134510-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 09:02:36.381661 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ez5NB/build/release/bin/kudu
/tmp/dist-test-task4ez5NB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859658134510-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859658134510-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859658134510-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859658134510-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:36989/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33121
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859658134510-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34093
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859658134510-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859658134510-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859658134510-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 09:02:36.495208 11844 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 09:02:36.495573 11844 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 09:02:36.495621 11844 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 09:02:36.497262 11844 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 09:02:36.497316 11844 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 09:02:36.497354 11844 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 09:02:36.499563 11844 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34093
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859658134510-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859658134510-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.98.20:36989/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859658134510-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859658134510-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33121
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11844
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859658134510-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 09:02:36.499801 11844 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 09:02:36.500070 11844 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 09:02:36.505959 11852 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:02:36.506058 11851 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:02:36.506325 11854 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:02:36.506707 11844 server_base.cc:1034] running on GCE node
I20260619 09:02:36.506939 11844 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 09:02:36.507195 11844 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 09:02:36.508368 11844 hybrid_clock.cc:648] HybridClock initialized: now 1781859756508345 us; error 33 us; skew 500 ppm
Jun 19 09:02:36 dist-test-slave-mtdr krb5kdc[11150](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859756, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 09:02:36.511652 11844 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 09:02:36.512169 11844 webserver.cc:460] Webserver started at http://127.0.98.2:43529/ using document root <none> and password file <none>
I20260619 09:02:36.512423 11844 fs_manager.cc:359] Metadata directory not provided
I20260619 09:02:36.512492 11844 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 09:02:36.512626 11844 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 09:02:36 dist-test-slave-mtdr krb5kdc[11150](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859756, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260619 09:02:36.527386 11844 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859658134510-392-0/minicluster-data/ts-1/data/instance:
uuid: "30a8073c322945a0ad50784f73418102"
format_stamp: "Formatted at 2026-06-19 09:02:36 on dist-test-slave-mtdr"
server_key: "0066c0ff0de2d8aa545a43a3d117eac1"
server_key_iv: "7dc4900570dce19423b5017a75dcc95f"
server_key_version: "kuduclusterkey@0"
I20260619 09:02:36.527827 11844 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859658134510-392-0/minicluster-data/ts-1/wal/instance:
uuid: "30a8073c322945a0ad50784f73418102"
format_stamp: "Formatted at 2026-06-19 09:02:36 on dist-test-slave-mtdr"
server_key: "0066c0ff0de2d8aa545a43a3d117eac1"
server_key_iv: "7dc4900570dce19423b5017a75dcc95f"
server_key_version: "kuduclusterkey@0"
Jun 19 09:02:36 dist-test-slave-mtdr krb5kdc[11150](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781859756, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 09:02:36.534201 11844 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.000s
I20260619 09:02:36.558748 11861 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 09:02:36.558959 11844 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 09:02:36.559052 11844 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859658134510-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859658134510-392-0/minicluster-data/ts-1/wal
uuid: "30a8073c322945a0ad50784f73418102"
format_stamp: "Formatted at 2026-06-19 09:02:36 on dist-test-slave-mtdr"
server_key: "0066c0ff0de2d8aa545a43a3d117eac1"
server_key_iv: "7dc4900570dce19423b5017a75dcc95f"
server_key_version: "kuduclusterkey@0"
I20260619 09:02:36.559149 11844 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859658134510-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859658134510-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859658134510-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 09:02:36.581527 11844 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 09:02:36.583355 11844 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 09:02:36.583480 11844 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 09:02:36.583674 11844 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 09:02:36.584008 11844 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 09:02:36.584043 11844 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 09:02:36.584072 11844 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 09:02:36.584121 11844 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 09:02:36.590791 11844 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38647
I20260619 09:02:36.590899 11974 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38647 every 8 connection(s)
I20260619 09:02:36.591437 11844 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859658134510-392-0/minicluster-data/ts-1/data/info.pb
Jun 19 09:02:36 dist-test-slave-mtdr krb5kdc[11150](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859756, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 09:02:36.600047 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ez5NB/build/release/bin/kudu as pid 11844
I20260619 09:02:36.600143 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859658134510-392-0/minicluster-data/ts-1/wal/instance
I20260619 09:02:36.600328 392 external_mini_cluster.cc:1428] Setting key 2a4cead527c8f2807e706989fb3dc0eb
I20260619 09:02:36.601188 11838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:02:36.592595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55839 (local address 127.0.98.62:33121)
0619 09:02:36.592641 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0619 09:02:36.592644 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 09:02:36.592849 (+ 205us) server_negotiation.cc:407] Connection header received
0619 09:02:36.593424 (+ 575us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:02:36.593426 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:02:36.593438 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:02:36.593515 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:02:36.594069 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:02:36.594918 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:02:36.595989 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:02:36.596276 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:02:36.596342 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:02:36.598346 (+ 2004us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:02:36.598347 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:02:36.598348 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:02:36.598355 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:02:36.599678 (+ 1323us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:02:36.600114 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:02:36.600116 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:02:36.600116 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:02:36.600173 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:02:36.600469 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:02:36.600470 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:02:36.600470 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:02:36.600606 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:02:36.600648 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0619 09:02:36.601008 (+ 360us) server_negotiation.cc:299] Negotiation successful
0619 09:02:36.601098 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260619 09:02:36.601680 11977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:02:36.592257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33121 (local address 127.0.98.2:55839)
0619 09:02:36.592728 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0619 09:02:36.592769 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 09:02:36.593291 (+ 522us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:02:36.593583 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:02:36.593587 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:02:36.593764 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0619 09:02:36.593994 (+ 230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:02:36.594000 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:02:36.595007 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:02:36.595008 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:02:36.595915 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:02:36.595918 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:02:36.596357 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:02:36.596358 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:02:36.596498 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:02:36.596995 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 09:02:36.597003 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 09:02:36.598250 (+ 1247us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 09:02:36.599750 (+ 1500us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:02:36.599757 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:02:36.599760 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:02:36.600022 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:02:36.600240 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:02:36.600240 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:02:36.600240 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:02:36.600391 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:02:36.600692 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 09:02:36.600698 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 09:02:36.600865 (+ 167us) client_negotiation.cc:769] Sending connection context
0619 09:02:36.601308 (+ 443us) client_negotiation.cc:240] Negotiation successful
0619 09:02:36.601462 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":399,"threads_started":1}
I20260619 09:02:36.602942 11975 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33121
I20260619 09:02:36.603035 11975 heartbeater.cc:464] Registering TS with master...
I20260619 09:02:36.603209 11975 heartbeater.cc:511] Master 127.0.98.62:33121 requested a full tablet report, sending...
I20260619 09:02:36.603549 11626 ts_manager.cc:194] Registered new tserver with Master: 30a8073c322945a0ad50784f73418102 (127.0.98.2:38647)
I20260619 09:02:36.603987 11626 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:55839
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859658134510-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859658134510-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 09:02:36.625707 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ez5NB/build/release/bin/kudu
/tmp/dist-test-task4ez5NB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859658134510-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859658134510-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859658134510-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859658134510-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:36989/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33121
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859658134510-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34093
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859658134510-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859658134510-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859658134510-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 09:02:36.711629 11981 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 09:02:36.711968 11981 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 09:02:36.712059 11981 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 09:02:36.713451 11981 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 09:02:36.713500 11981 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 09:02:36.713534 11981 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 09:02:36.715340 11981 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34093
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859658134510-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859658134510-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.98.20:36989/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859658134510-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859658134510-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33121
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11981
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859658134510-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 09:02:36.715587 11981 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 09:02:36.715837 11981 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 09:02:36.721292 11989 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:02:36.721633 11987 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:02:36.722774 11981 server_base.cc:1034] running on GCE node
W20260619 09:02:36.727582 11986 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:02:36.727840 11981 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 09:02:36.728104 11981 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 09:02:36.729264 11981 hybrid_clock.cc:648] HybridClock initialized: now 1781859756729244 us; error 35 us; skew 500 ppm
Jun 19 09:02:36 dist-test-slave-mtdr krb5kdc[11150](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859756, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 09:02:36.732191 11981 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 09:02:36.732610 11981 webserver.cc:460] Webserver started at http://127.0.98.3:41849/ using document root <none> and password file <none>
I20260619 09:02:36.732825 11981 fs_manager.cc:359] Metadata directory not provided
I20260619 09:02:36.732873 11981 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 09:02:36.733042 11981 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 09:02:36 dist-test-slave-mtdr krb5kdc[11150](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859756, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260619 09:02:36.756054 11981 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859658134510-392-0/minicluster-data/ts-2/data/instance:
uuid: "a52a6e04c6844a4897ab7c83de517e27"
format_stamp: "Formatted at 2026-06-19 09:02:36 on dist-test-slave-mtdr"
server_key: "d0fe80aecfb0ef2832bbb2dda51dad7a"
server_key_iv: "13757f4bce29c21113fa023a6a72c089"
server_key_version: "kuduclusterkey@0"
I20260619 09:02:36.756606 11981 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859658134510-392-0/minicluster-data/ts-2/wal/instance:
uuid: "a52a6e04c6844a4897ab7c83de517e27"
format_stamp: "Formatted at 2026-06-19 09:02:36 on dist-test-slave-mtdr"
server_key: "d0fe80aecfb0ef2832bbb2dda51dad7a"
server_key_iv: "13757f4bce29c21113fa023a6a72c089"
server_key_version: "kuduclusterkey@0"
I20260619 09:02:36.759414 11981 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.001s
I20260619 09:02:36.778983 11996 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 09:02:36.779263 11981 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.000s sys 0.001s
I20260619 09:02:36.779409 11981 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859658134510-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859658134510-392-0/minicluster-data/ts-2/wal
uuid: "a52a6e04c6844a4897ab7c83de517e27"
format_stamp: "Formatted at 2026-06-19 09:02:36 on dist-test-slave-mtdr"
server_key: "d0fe80aecfb0ef2832bbb2dda51dad7a"
server_key_iv: "13757f4bce29c21113fa023a6a72c089"
server_key_version: "kuduclusterkey@0"
I20260619 09:02:36.779585 11981 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859658134510-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859658134510-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859658134510-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 09:02:36.790711 11981 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 09:02:36.793198 11981 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 09:02:36.793387 11981 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 09:02:36.793628 11981 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 09:02:36.794029 11981 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 09:02:36.794129 11981 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 09:02:36.794201 11981 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 09:02:36.794255 11981 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 09:02:36.800618 11981 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38969
I20260619 09:02:36.801065 11981 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859658134510-392-0/minicluster-data/ts-2/data/info.pb
I20260619 09:02:36.803211 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ez5NB/build/release/bin/kudu as pid 11981
I20260619 09:02:36.803304 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859658134510-392-0/minicluster-data/ts-2/wal/instance
I20260619 09:02:36.803524 392 external_mini_cluster.cc:1428] Setting key fad4aa84e59ac502189198f78f378750
I20260619 09:02:36.804196 12109 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38969 every 8 connection(s)
Jun 19 09:02:36 dist-test-slave-mtdr krb5kdc[11150](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859756, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 09:02:36.813031 11838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:02:36.804790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38149 (local address 127.0.98.62:33121)
0619 09:02:36.804831 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0619 09:02:36.804833 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:02:36.805079 (+ 246us) server_negotiation.cc:407] Connection header received
0619 09:02:36.805545 (+ 466us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:02:36.805546 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:02:36.805557 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:02:36.805607 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:02:36.806230 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:02:36.807018 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:02:36.808034 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:02:36.808219 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:02:36.808274 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:02:36.810436 (+ 2162us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:02:36.810437 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:02:36.810437 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:02:36.810444 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:02:36.811792 (+ 1348us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:02:36.812156 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:02:36.812158 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:02:36.812158 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:02:36.812207 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:02:36.812415 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:02:36.812415 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:02:36.812416 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:02:36.812526 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:02:36.812584 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0619 09:02:36.812868 (+ 284us) server_negotiation.cc:299] Negotiation successful
0619 09:02:36.812943 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260619 09:02:36.813189 12112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:02:36.804782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33121 (local address 127.0.98.3:38149)
0619 09:02:36.804952 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0619 09:02:36.804994 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 09:02:36.805453 (+ 459us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:02:36.805674 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:02:36.805677 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:02:36.805853 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0619 09:02:36.806118 (+ 265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:02:36.806125 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:02:36.807150 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:02:36.807150 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:02:36.807949 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:02:36.807951 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:02:36.808290 (+ 339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:02:36.808290 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:02:36.808405 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:02:36.808905 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 09:02:36.808914 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0619 09:02:36.810312 (+ 1398us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 09:02:36.811874 (+ 1562us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:02:36.811881 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:02:36.811883 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:02:36.812092 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:02:36.812259 (+ 167us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:02:36.812260 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:02:36.812260 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:02:36.812362 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:02:36.812611 (+ 249us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 09:02:36.812617 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 09:02:36.812752 (+ 135us) client_negotiation.cc:769] Sending connection context
0619 09:02:36.812838 (+ 86us) client_negotiation.cc:240] Negotiation successful
0619 09:02:36.812980 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":90,"threads_started":1}
I20260619 09:02:36.813614 12110 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33121
I20260619 09:02:36.813702 12110 heartbeater.cc:464] Registering TS with master...
I20260619 09:02:36.813877 12110 heartbeater.cc:511] Master 127.0.98.62:33121 requested a full tablet report, sending...
I20260619 09:02:36.814289 11626 ts_manager.cc:194] Registered new tserver with Master: a52a6e04c6844a4897ab7c83de517e27 (127.0.98.3:38969)
I20260619 09:02:36.814683 11626 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:38149
I20260619 09:02:36.815295 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 09:02:36.825079 11838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:02:36.819070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50400 (local address 127.0.98.62:33121)
0619 09:02:36.819108 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0619 09:02:36.819110 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:02:36.819383 (+ 273us) server_negotiation.cc:407] Connection header received
0619 09:02:36.819437 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:02:36.819438 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:02:36.819447 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:02:36.819491 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:02:36.820147 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:02:36.820707 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:02:36.821406 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:02:36.821586 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:02:36.821637 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:02:36.822722 (+ 1085us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:02:36.822723 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:02:36.822724 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:02:36.822730 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:02:36.823977 (+ 1247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:02:36.824323 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:02:36.824324 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:02:36.824324 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:02:36.824375 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:02:36.824553 (+ 178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:02:36.824553 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:02:36.824554 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:02:36.824657 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:02:36.824723 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0619 09:02:36.824928 (+ 205us) server_negotiation.cc:299] Negotiation successful
0619 09:02:36.824997 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260619 09:02:36.827045 11626 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50400:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 09:02:36.829205 11626 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 09:02:36.836038 12125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:02:36.831523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38647 (local address 127.0.0.1:60828)
0619 09:02:36.832118 (+ 595us) negotiation.cc:107] Waiting for socket to connect
0619 09:02:36.832126 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0619 09:02:36.832171 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:02:36.832460 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:02:36.832463 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:02:36.832465 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 09:02:36.832534 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:02:36.832536 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:02:36.833839 (+ 1303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:02:36.833840 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:02:36.834749 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:02:36.834751 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:02:36.835765 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:02:36.835766 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:02:36.835843 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:02:36.835855 (+ 12us) client_negotiation.cc:769] Sending connection context
0619 09:02:36.835910 (+ 55us) client_negotiation.cc:240] Negotiation successful
0619 09:02:36.835953 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":562,"thread_start_us":84,"threads_started":1}
I20260619 09:02:36.836308 12124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:02:36.831642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60828 (local address 127.0.98.2:38647)
0619 09:02:36.831772 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0619 09:02:36.831777 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 09:02:36.832180 (+ 403us) server_negotiation.cc:407] Connection header received
0619 09:02:36.832223 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:02:36.832226 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:02:36.832241 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:02:36.832312 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 09:02:36.832626 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:02:36.833753 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:02:36.834865 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:02:36.835598 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:02:36.835666 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:02:36.835735 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0619 09:02:36.836188 (+ 453us) server_negotiation.cc:299] Negotiation successful
0619 09:02:36.836228 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98,"thread_start_us":67,"threads_started":1}
I20260619 09:02:36.836529 12126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:02:36.831854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40270 (local address 127.0.98.1:39133)
0619 09:02:36.831982 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0619 09:02:36.831987 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 09:02:36.832294 (+ 307us) server_negotiation.cc:407] Connection header received
0619 09:02:36.832341 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:02:36.832343 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:02:36.832359 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:02:36.832442 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 09:02:36.832652 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:02:36.833725 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:02:36.835262 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:02:36.835992 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:02:36.836057 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:02:36.836125 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0619 09:02:36.836423 (+ 298us) server_negotiation.cc:299] Negotiation successful
0619 09:02:36.836460 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89,"thread_start_us":59,"threads_started":1}
I20260619 09:02:36.836359 12123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:02:36.831730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39133 (local address 127.0.0.1:40270)
0619 09:02:36.832235 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0619 09:02:36.832242 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0619 09:02:36.832284 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:02:36.832542 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:02:36.832543 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:02:36.832544 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 09:02:36.832577 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:02:36.832578 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:02:36.833816 (+ 1238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:02:36.833817 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:02:36.835152 (+ 1335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:02:36.835154 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:02:36.836158 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:02:36.836158 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:02:36.836214 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:02:36.836218 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 09:02:36.836258 (+ 40us) client_negotiation.cc:240] Negotiation successful
0619 09:02:36.836292 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":493,"mutex_wait_us":31,"thread_start_us":57,"threads_started":1}
I20260619 09:02:36.837153 11909 tablet_service.cc:1467] Processing CreateTablet for tablet c00cf811e404442aaab07127e6baeefb (DEFAULT_TABLE table=test-table [id=d33901190d8b4dda91ce05bf57dba5ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 09:02:36.837535 11909 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c00cf811e404442aaab07127e6baeefb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 09:02:36.837594 11752 tablet_service.cc:1467] Processing CreateTablet for tablet c00cf811e404442aaab07127e6baeefb (DEFAULT_TABLE table=test-table [id=d33901190d8b4dda91ce05bf57dba5ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 09:02:36.837915 11752 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c00cf811e404442aaab07127e6baeefb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 09:02:36.838740 12122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:02:36.831515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38969 (local address 127.0.0.1:59198)
0619 09:02:36.831884 (+ 369us) negotiation.cc:107] Waiting for socket to connect
0619 09:02:36.831925 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 09:02:36.831996 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:02:36.834916 (+ 2920us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:02:36.834917 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:02:36.834920 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 09:02:36.834956 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:02:36.834957 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:02:36.836273 (+ 1316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:02:36.836274 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:02:36.837674 (+ 1400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:02:36.837676 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:02:36.838550 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:02:36.838551 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:02:36.838614 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:02:36.838618 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 09:02:36.838645 (+ 27us) client_negotiation.cc:240] Negotiation successful
0619 09:02:36.838678 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":81,"threads_started":1}
I20260619 09:02:36.839105 12127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:02:36.831915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59198 (local address 127.0.98.3:38969)
0619 09:02:36.832076 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0619 09:02:36.832081 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 09:02:36.832091 (+ 10us) server_negotiation.cc:407] Connection header received
0619 09:02:36.832126 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:02:36.832128 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:02:36.832144 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:02:36.835023 (+ 2879us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 09:02:36.835087 (+ 64us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:02:36.836048 (+ 961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:02:36.837783 (+ 1735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:02:36.838444 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:02:36.838897 (+ 453us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:02:36.838976 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0619 09:02:36.839001 (+ 25us) server_negotiation.cc:299] Negotiation successful
0619 09:02:36.839035 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":86,"threads_started":1}
I20260619 09:02:36.839738 12044 tablet_service.cc:1467] Processing CreateTablet for tablet c00cf811e404442aaab07127e6baeefb (DEFAULT_TABLE table=test-table [id=d33901190d8b4dda91ce05bf57dba5ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 09:02:36.840013 12044 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c00cf811e404442aaab07127e6baeefb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 09:02:36.841535 12129 tablet_bootstrap.cc:492] T c00cf811e404442aaab07127e6baeefb P cd3b555594c449ceb5e0ce4a7f38d4f8: Bootstrap starting.
I20260619 09:02:36.842509 12129 tablet_bootstrap.cc:654] T c00cf811e404442aaab07127e6baeefb P cd3b555594c449ceb5e0ce4a7f38d4f8: Neither blocks nor log segments found. Creating new log.
I20260619 09:02:36.842865 12129 log.cc:824] T c00cf811e404442aaab07127e6baeefb P cd3b555594c449ceb5e0ce4a7f38d4f8: Log is configured to *not* fsync() on all Append() calls
I20260619 09:02:36.842981 12128 tablet_bootstrap.cc:492] T c00cf811e404442aaab07127e6baeefb P 30a8073c322945a0ad50784f73418102: Bootstrap starting.
I20260619 09:02:36.843421 12130 tablet_bootstrap.cc:492] T c00cf811e404442aaab07127e6baeefb P a52a6e04c6844a4897ab7c83de517e27: Bootstrap starting.
I20260619 09:02:36.844014 12128 tablet_bootstrap.cc:654] T c00cf811e404442aaab07127e6baeefb P 30a8073c322945a0ad50784f73418102: Neither blocks nor log segments found. Creating new log.
I20260619 09:02:36.844274 12130 tablet_bootstrap.cc:654] T c00cf811e404442aaab07127e6baeefb P a52a6e04c6844a4897ab7c83de517e27: Neither blocks nor log segments found. Creating new log.
I20260619 09:02:36.844280 12128 log.cc:824] T c00cf811e404442aaab07127e6baeefb P 30a8073c322945a0ad50784f73418102: Log is configured to *not* fsync() on all Append() calls
I20260619 09:02:36.844549 12130 log.cc:824] T c00cf811e404442aaab07127e6baeefb P a52a6e04c6844a4897ab7c83de517e27: Log is configured to *not* fsync() on all Append() calls
I20260619 09:02:36.845077 12128 tablet_bootstrap.cc:492] T c00cf811e404442aaab07127e6baeefb P 30a8073c322945a0ad50784f73418102: No bootstrap required, opened a new log
I20260619 09:02:36.845142 12128 ts_tablet_manager.cc:1397] T c00cf811e404442aaab07127e6baeefb P 30a8073c322945a0ad50784f73418102: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260619 09:02:36.847061 12128 raft_consensus.cc:348] T c00cf811e404442aaab07127e6baeefb P 30a8073c322945a0ad50784f73418102 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a52a6e04c6844a4897ab7c83de517e27" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38969 } } peers { permanent_uuid: "30a8073c322945a0ad50784f73418102" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38647 } } peers { permanent_uuid: "cd3b555594c449ceb5e0ce4a7f38d4f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39133 } }
I20260619 09:02:36.847250 12130 tablet_bootstrap.cc:492] T c00cf811e404442aaab07127e6baeefb P a52a6e04c6844a4897ab7c83de517e27: No bootstrap required, opened a new log
I20260619 09:02:36.847272 12128 raft_consensus.cc:374] T c00cf811e404442aaab07127e6baeefb P 30a8073c322945a0ad50784f73418102 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 09:02:36.847330 12130 ts_tablet_manager.cc:1397] T c00cf811e404442aaab07127e6baeefb P a52a6e04c6844a4897ab7c83de517e27: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260619 09:02:36.847335 12128 raft_consensus.cc:729] T c00cf811e404442aaab07127e6baeefb P 30a8073c322945a0ad50784f73418102 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30a8073c322945a0ad50784f73418102, State: Initialized, Role: FOLLOWER
I20260619 09:02:36.847460 12128 consensus_queue.cc:260] T c00cf811e404442aaab07127e6baeefb P 30a8073c322945a0ad50784f73418102 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a52a6e04c6844a4897ab7c83de517e27" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38969 } } peers { permanent_uuid: "30a8073c322945a0ad50784f73418102" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38647 } } peers { permanent_uuid: "cd3b555594c449ceb5e0ce4a7f38d4f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39133 } }
I20260619 09:02:36.847713 12128 ts_tablet_manager.cc:1428] T c00cf811e404442aaab07127e6baeefb P 30a8073c322945a0ad50784f73418102: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260619 09:02:36.847834 11975 heartbeater.cc:503] Master 127.0.98.62:33121 was elected leader, sending a full tablet report...
I20260619 09:02:36.848822 12130 raft_consensus.cc:348] T c00cf811e404442aaab07127e6baeefb P a52a6e04c6844a4897ab7c83de517e27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a52a6e04c6844a4897ab7c83de517e27" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38969 } } peers { permanent_uuid: "30a8073c322945a0ad50784f73418102" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38647 } } peers { permanent_uuid: "cd3b555594c449ceb5e0ce4a7f38d4f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39133 } }
I20260619 09:02:36.848940 12130 raft_consensus.cc:374] T c00cf811e404442aaab07127e6baeefb P a52a6e04c6844a4897ab7c83de517e27 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 09:02:36.848970 12130 raft_consensus.cc:729] T c00cf811e404442aaab07127e6baeefb P a52a6e04c6844a4897ab7c83de517e27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a52a6e04c6844a4897ab7c83de517e27, State: Initialized, Role: FOLLOWER
I20260619 09:02:36.849080 12130 consensus_queue.cc:260] T c00cf811e404442aaab07127e6baeefb P a52a6e04c6844a4897ab7c83de517e27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a52a6e04c6844a4897ab7c83de517e27" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38969 } } peers { permanent_uuid: "30a8073c322945a0ad50784f73418102" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38647 } } peers { permanent_uuid: "cd3b555594c449ceb5e0ce4a7f38d4f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39133 } }
I20260619 09:02:36.849291 12130 ts_tablet_manager.cc:1428] T c00cf811e404442aaab07127e6baeefb P a52a6e04c6844a4897ab7c83de517e27: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 09:02:36.849334 12110 heartbeater.cc:503] Master 127.0.98.62:33121 was elected leader, sending a full tablet report...
I20260619 09:02:36.851480 12129 tablet_bootstrap.cc:492] T c00cf811e404442aaab07127e6baeefb P cd3b555594c449ceb5e0ce4a7f38d4f8: No bootstrap required, opened a new log
I20260619 09:02:36.851557 12129 ts_tablet_manager.cc:1397] T c00cf811e404442aaab07127e6baeefb P cd3b555594c449ceb5e0ce4a7f38d4f8: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.001s
I20260619 09:02:36.853539 12129 raft_consensus.cc:348] T c00cf811e404442aaab07127e6baeefb P cd3b555594c449ceb5e0ce4a7f38d4f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a52a6e04c6844a4897ab7c83de517e27" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38969 } } peers { permanent_uuid: "30a8073c322945a0ad50784f73418102" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38647 } } peers { permanent_uuid: "cd3b555594c449ceb5e0ce4a7f38d4f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39133 } }
I20260619 09:02:36.853711 12129 raft_consensus.cc:374] T c00cf811e404442aaab07127e6baeefb P cd3b555594c449ceb5e0ce4a7f38d4f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 09:02:36.853752 12129 raft_consensus.cc:729] T c00cf811e404442aaab07127e6baeefb P cd3b555594c449ceb5e0ce4a7f38d4f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd3b555594c449ceb5e0ce4a7f38d4f8, State: Initialized, Role: FOLLOWER
I20260619 09:02:36.853878 12129 consensus_queue.cc:260] T c00cf811e404442aaab07127e6baeefb P cd3b555594c449ceb5e0ce4a7f38d4f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a52a6e04c6844a4897ab7c83de517e27" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38969 } } peers { permanent_uuid: "30a8073c322945a0ad50784f73418102" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38647 } } peers { permanent_uuid: "cd3b555594c449ceb5e0ce4a7f38d4f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39133 } }
I20260619 09:02:36.854102 12129 ts_tablet_manager.cc:1428] T c00cf811e404442aaab07127e6baeefb P cd3b555594c449ceb5e0ce4a7f38d4f8: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260619 09:02:36.854118 11836 heartbeater.cc:503] Master 127.0.98.62:33121 was elected leader, sending a full tablet report...
I20260619 09:02:36.862712 12134 raft_consensus.cc:482] T c00cf811e404442aaab07127e6baeefb P 30a8073c322945a0ad50784f73418102 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 09:02:36.862903 12134 raft_consensus.cc:504] T c00cf811e404442aaab07127e6baeefb P 30a8073c322945a0ad50784f73418102 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a52a6e04c6844a4897ab7c83de517e27" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38969 } } peers { permanent_uuid: "30a8073c322945a0ad50784f73418102" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38647 } } peers { permanent_uuid: "cd3b555594c449ceb5e0ce4a7f38d4f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39133 } }
I20260619 09:02:36.863250 12134 leader_election.cc:288] T c00cf811e404442aaab07127e6baeefb P 30a8073c322945a0ad50784f73418102 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a52a6e04c6844a4897ab7c83de517e27 (127.0.98.3:38969), cd3b555594c449ceb5e0ce4a7f38d4f8 (127.0.98.1:39133)
I20260619 09:02:36.866758 12127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:02:36.863669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34683 (local address 127.0.98.3:38969)
0619 09:02:36.863720 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0619 09:02:36.863722 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:02:36.863731 (+ 9us) server_negotiation.cc:407] Connection header received
0619 09:02:36.863757 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:02:36.863758 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:02:36.863769 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:02:36.863826 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 09:02:36.864512 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:02:36.865037 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:02:36.866088 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:02:36.866455 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:02:36.866514 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:02:36.866524 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0619 09:02:36.866659 (+ 135us) server_negotiation.cc:299] Negotiation successful
0619 09:02:36.866692 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260619 09:02:36.866760 11977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:02:36.863372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38969 (local address 127.0.98.2:34683)
0619 09:02:36.863624 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0619 09:02:36.863636 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 09:02:36.863683 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:02:36.863830 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:02:36.863831 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:02:36.863832 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 09:02:36.864412 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:02:36.864415 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:02:36.865138 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:02:36.865139 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:02:36.866005 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:02:36.866008 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:02:36.866524 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:02:36.866525 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:02:36.866601 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:02:36.866605 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 09:02:36.866648 (+ 43us) client_negotiation.cc:240] Negotiation successful
0619 09:02:36.866692 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":231}
I20260619 09:02:36.867298 12064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c00cf811e404442aaab07127e6baeefb" candidate_uuid: "30a8073c322945a0ad50784f73418102" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a52a6e04c6844a4897ab7c83de517e27" is_pre_election: true
I20260619 09:02:36.867439 12064 raft_consensus.cc:2454] T c00cf811e404442aaab07127e6baeefb P a52a6e04c6844a4897ab7c83de517e27 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30a8073c322945a0ad50784f73418102 in term 0.
I20260619 09:02:36.867633 11863 leader_election.cc:302] T c00cf811e404442aaab07127e6baeefb P 30a8073c322945a0ad50784f73418102 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30a8073c322945a0ad50784f73418102, a52a6e04c6844a4897ab7c83de517e27; no voters:
I20260619 09:02:36.867853 12134 raft_consensus.cc:2781] T c00cf811e404442aaab07127e6baeefb P 30a8073c322945a0ad50784f73418102 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 09:02:36.867924 12134 raft_consensus.cc:482] T c00cf811e404442aaab07127e6baeefb P 30a8073c322945a0ad50784f73418102 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 09:02:36.867946 12134 raft_consensus.cc:3037] T c00cf811e404442aaab07127e6baeefb P 30a8073c322945a0ad50784f73418102 [term 0 FOLLOWER]: Advancing to term 1
I20260619 09:02:36.868743 12134 raft_consensus.cc:504] T c00cf811e404442aaab07127e6baeefb P 30a8073c322945a0ad50784f73418102 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a52a6e04c6844a4897ab7c83de517e27" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38969 } } peers { permanent_uuid: "30a8073c322945a0ad50784f73418102" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38647 } } peers { permanent_uuid: "cd3b555594c449ceb5e0ce4a7f38d4f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39133 } }
I20260619 09:02:36.868971 12064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c00cf811e404442aaab07127e6baeefb" candidate_uuid: "30a8073c322945a0ad50784f73418102" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a52a6e04c6844a4897ab7c83de517e27"
I20260619 09:02:36.869050 12064 raft_consensus.cc:3037] T c00cf811e404442aaab07127e6baeefb P a52a6e04c6844a4897ab7c83de517e27 [term 0 FOLLOWER]: Advancing to term 1
I20260619 09:02:36.869714 11977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:02:36.863474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39133 (local address 127.0.98.2:57111)
0619 09:02:36.866849 (+ 3375us) negotiation.cc:107] Waiting for socket to connect
0619 09:02:36.866859 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 09:02:36.866916 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:02:36.867066 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:02:36.867067 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:02:36.867068 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 09:02:36.867099 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:02:36.867101 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:02:36.867865 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:02:36.867866 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:02:36.868810 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:02:36.868812 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:02:36.869507 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:02:36.869508 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:02:36.869566 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:02:36.869569 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 09:02:36.869611 (+ 42us) client_negotiation.cc:240] Negotiation successful
0619 09:02:36.869645 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3364,"thread_start_us":71,"threads_started":1}
I20260619 09:02:36.869838 12064 raft_consensus.cc:2454] T c00cf811e404442aaab07127e6baeefb P a52a6e04c6844a4897ab7c83de517e27 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30a8073c322945a0ad50784f73418102 in term 1.
I20260619 09:02:36.869886 12126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:02:36.863716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57111 (local address 127.0.98.1:39133)
0619 09:02:36.863755 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0619 09:02:36.863757 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:02:36.866967 (+ 3210us) server_negotiation.cc:407] Connection header received
0619 09:02:36.866992 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:02:36.866993 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:02:36.867001 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:02:36.867045 (+ 44us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 09:02:36.867165 (+ 120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:02:36.867784 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:02:36.868904 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:02:36.869406 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:02:36.869468 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:02:36.869478 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0619 09:02:36.869791 (+ 313us) server_negotiation.cc:299] Negotiation successful
0619 09:02:36.869828 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260619 09:02:36.870005 11863 leader_election.cc:302] T c00cf811e404442aaab07127e6baeefb P 30a8073c322945a0ad50784f73418102 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30a8073c322945a0ad50784f73418102, a52a6e04c6844a4897ab7c83de517e27; no voters:
I20260619 09:02:36.870190 12138 raft_consensus.cc:2781] T c00cf811e404442aaab07127e6baeefb P 30a8073c322945a0ad50784f73418102 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 09:02:36.870306 12138 raft_consensus.cc:686] T c00cf811e404442aaab07127e6baeefb P 30a8073c322945a0ad50784f73418102 [term 1 LEADER]: Becoming Leader. State: Replica: 30a8073c322945a0ad50784f73418102, State: Running, Role: LEADER
I20260619 09:02:36.870333 11790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c00cf811e404442aaab07127e6baeefb" candidate_uuid: "30a8073c322945a0ad50784f73418102" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd3b555594c449ceb5e0ce4a7f38d4f8" is_pre_election: true
I20260619 09:02:36.870399 12138 consensus_queue.cc:237] T c00cf811e404442aaab07127e6baeefb P 30a8073c322945a0ad50784f73418102 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a52a6e04c6844a4897ab7c83de517e27" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38969 } } peers { permanent_uuid: "30a8073c322945a0ad50784f73418102" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38647 } } peers { permanent_uuid: "cd3b555594c449ceb5e0ce4a7f38d4f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39133 } }
I20260619 09:02:36.870452 11790 raft_consensus.cc:2454] T c00cf811e404442aaab07127e6baeefb P cd3b555594c449ceb5e0ce4a7f38d4f8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30a8073c322945a0ad50784f73418102 in term 0.
I20260619 09:02:36.871021 12134 leader_election.cc:288] T c00cf811e404442aaab07127e6baeefb P 30a8073c322945a0ad50784f73418102 [CANDIDATE]: Term 1 election: Requested vote from peers a52a6e04c6844a4897ab7c83de517e27 (127.0.98.3:38969), cd3b555594c449ceb5e0ce4a7f38d4f8 (127.0.98.1:39133)
I20260619 09:02:36.871235 11790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c00cf811e404442aaab07127e6baeefb" candidate_uuid: "30a8073c322945a0ad50784f73418102" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd3b555594c449ceb5e0ce4a7f38d4f8"
I20260619 09:02:36.871304 11790 raft_consensus.cc:3037] T c00cf811e404442aaab07127e6baeefb P cd3b555594c449ceb5e0ce4a7f38d4f8 [term 0 FOLLOWER]: Advancing to term 1
I20260619 09:02:36.871217 11625 catalog_manager.cc:5495] T c00cf811e404442aaab07127e6baeefb P 30a8073c322945a0ad50784f73418102 reported cstate change: term changed from 0 to 1, leader changed from <none> to 30a8073c322945a0ad50784f73418102 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "30a8073c322945a0ad50784f73418102" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a52a6e04c6844a4897ab7c83de517e27" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38969 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30a8073c322945a0ad50784f73418102" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38647 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cd3b555594c449ceb5e0ce4a7f38d4f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39133 } health_report { overall_health: UNKNOWN } } }
I20260619 09:02:36.871940 11790 raft_consensus.cc:2454] T c00cf811e404442aaab07127e6baeefb P cd3b555594c449ceb5e0ce4a7f38d4f8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30a8073c322945a0ad50784f73418102 in term 1.
I20260619 09:02:36.881937 12124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:02:36.879746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60834 (local address 127.0.98.2:38647)
0619 09:02:36.879780 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0619 09:02:36.879782 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:02:36.879790 (+ 8us) server_negotiation.cc:407] Connection header received
0619 09:02:36.879860 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:02:36.879861 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:02:36.879888 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:02:36.879951 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 09:02:36.880062 (+ 111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:02:36.880509 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:02:36.881219 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:02:36.881429 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:02:36.881482 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:02:36.881625 (+ 143us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 09:02:36.881719 (+ 94us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 09:02:36.881786 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0619 09:02:36.881822 (+ 36us) server_negotiation.cc:299] Negotiation successful
0619 09:02:36.881840 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260619 09:02:36.883811 11790 raft_consensus.cc:1261] T c00cf811e404442aaab07127e6baeefb P cd3b555594c449ceb5e0ce4a7f38d4f8 [term 1 FOLLOWER]: Refusing update from remote peer 30a8073c322945a0ad50784f73418102: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 09:02:36.884357 12064 raft_consensus.cc:1261] T c00cf811e404442aaab07127e6baeefb P a52a6e04c6844a4897ab7c83de517e27 [term 1 FOLLOWER]: Refusing update from remote peer 30a8073c322945a0ad50784f73418102: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 09:02:36.884603 12134 consensus_queue.cc:1035] T c00cf811e404442aaab07127e6baeefb P 30a8073c322945a0ad50784f73418102 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd3b555594c449ceb5e0ce4a7f38d4f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 09:02:36.884774 12139 consensus_queue.cc:1035] T c00cf811e404442aaab07127e6baeefb P 30a8073c322945a0ad50784f73418102 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a52a6e04c6844a4897ab7c83de517e27" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 09:02:36.888849 12145 mvcc.cc:205] Tried to move back new op lower bound from 7298497564192514048 to 7298497564141228032. Current Snapshot: MvccSnapshot[applied={T|T < 7298497564192514048}]
I20260619 09:02:36.889554 12143 mvcc.cc:205] Tried to move back new op lower bound from 7298497564192514048 to 7298497564141228032. Current Snapshot: MvccSnapshot[applied={T|T < 7298497564192514048}]
I20260619 09:02:36.891355 11626 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50400:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 09:02:36.891440 11626 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50400:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 09:02:36.892185 11626 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 63e23856a1e04f25aacd64df8898860d: Sending DeleteTablet for 3 replicas of tablet c00cf811e404442aaab07127e6baeefb
I20260619 09:02:36.892567 12044 tablet_service.cc:1514] Processing DeleteTablet for tablet c00cf811e404442aaab07127e6baeefb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 09:02:36 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59198
I20260619 09:02:36.892753 11752 tablet_service.cc:1514] Processing DeleteTablet for tablet c00cf811e404442aaab07127e6baeefb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 09:02:36 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40270
I20260619 09:02:36.893105 11909 tablet_service.cc:1514] Processing DeleteTablet for tablet c00cf811e404442aaab07127e6baeefb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 09:02:36 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60828
I20260619 09:02:36.893615 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ez5NB/build/release/bin/kudu with pid 11707
I20260619 09:02:36.898962 12153 tablet_replica.cc:330] T c00cf811e404442aaab07127e6baeefb P 30a8073c322945a0ad50784f73418102: stopping tablet replica
I20260619 09:02:36.899104 12151 tablet_replica.cc:330] T c00cf811e404442aaab07127e6baeefb P a52a6e04c6844a4897ab7c83de517e27: stopping tablet replica
I20260619 09:02:36.899261 12151 raft_consensus.cc:2229] T c00cf811e404442aaab07127e6baeefb P a52a6e04c6844a4897ab7c83de517e27 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 09:02:36.899395 12153 raft_consensus.cc:2229] T c00cf811e404442aaab07127e6baeefb P 30a8073c322945a0ad50784f73418102 [term 1 LEADER]: Raft consensus shutting down.
I20260619 09:02:36.899555 12151 raft_consensus.cc:2258] T c00cf811e404442aaab07127e6baeefb P a52a6e04c6844a4897ab7c83de517e27 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 09:02:36.899721 12153 raft_consensus.cc:2258] T c00cf811e404442aaab07127e6baeefb P 30a8073c322945a0ad50784f73418102 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 09:02:36.900315 12151 ts_tablet_manager.cc:1905] T c00cf811e404442aaab07127e6baeefb P a52a6e04c6844a4897ab7c83de517e27: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 09:02:36.900745 12153 ts_tablet_manager.cc:1905] T c00cf811e404442aaab07127e6baeefb P 30a8073c322945a0ad50784f73418102: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 09:02:36.905742 12153 ts_tablet_manager.cc:1918] T c00cf811e404442aaab07127e6baeefb P 30a8073c322945a0ad50784f73418102: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 09:02:36.907356 12151 ts_tablet_manager.cc:1918] T c00cf811e404442aaab07127e6baeefb P a52a6e04c6844a4897ab7c83de517e27: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 09:02:36.907490 12151 log.cc:1196] T c00cf811e404442aaab07127e6baeefb P a52a6e04c6844a4897ab7c83de517e27: Deleting WAL directory at /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859658134510-392-0/minicluster-data/ts-2/wal/wals/c00cf811e404442aaab07127e6baeefb
I20260619 09:02:36.907595 12153 log.cc:1196] T c00cf811e404442aaab07127e6baeefb P 30a8073c322945a0ad50784f73418102: Deleting WAL directory at /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781859658134510-392-0/minicluster-data/ts-1/wal/wals/c00cf811e404442aaab07127e6baeefb
I20260619 09:02:36.908020 12151 ts_tablet_manager.cc:1939] T c00cf811e404442aaab07127e6baeefb P a52a6e04c6844a4897ab7c83de517e27: Deleting consensus metadata
I20260619 09:02:36.908313 12153 ts_tablet_manager.cc:1939] T c00cf811e404442aaab07127e6baeefb P 30a8073c322945a0ad50784f73418102: Deleting consensus metadata
I20260619 09:02:36.908793 11611 catalog_manager.cc:4841] TS a52a6e04c6844a4897ab7c83de517e27 (127.0.98.3:38969): tablet c00cf811e404442aaab07127e6baeefb (table test-table [id=d33901190d8b4dda91ce05bf57dba5ea]) successfully deleted
I20260619 09:02:36.909302 11613 catalog_manager.cc:4841] TS 30a8073c322945a0ad50784f73418102 (127.0.98.2:38647): tablet c00cf811e404442aaab07127e6baeefb (table test-table [id=d33901190d8b4dda91ce05bf57dba5ea]) successfully deleted
W20260619 09:02:36.909570 11611 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:39133 (error 108)
I20260619 09:02:36.910441 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ez5NB/build/release/bin/kudu with pid 11844
I20260619 09:02:36.910458 12122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:02:36.910313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39133 (local address 127.0.0.1:40282)
0619 09:02:36.910368 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0619 09:02:36.910423 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260619 09:02:36.910989 11611 catalog_manager.cc:4584] TS cd3b555594c449ceb5e0ce4a7f38d4f8 (127.0.98.1:39133): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c00cf811e404442aaab07127e6baeefb: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39133: connect: Connection refused (error 111)
I20260619 09:02:36.916013 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ez5NB/build/release/bin/kudu with pid 11981
I20260619 09:02:36.920547 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ez5NB/build/release/bin/kudu with pid 11594
I20260619 09:02:36.924604 392 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260619 09:02:37.258677 392 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260619 09:02:37.258785 392 mini_ranger.cc:66] Stopping Ranger...
I20260619 09:02:37.592669 392 mini_ranger.cc:68] Stopped Ranger
2026-06-19 09:02:37.592 UTC [11180] LOG: received smart shutdown request
2026-06-19 09:02:37.595 UTC [11180] LOG: background worker "logical replication launcher" (PID 11189) exited with exit code 1
2026-06-19 09:02:37.596 UTC [11184] LOG: shutting down
2026-06-19 09:02:37.751 UTC [11180] LOG: database system is shut down
2026-06-19T09:02:37Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (42946 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859658134510-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 09:02:37 dist-test-slave-mtdr krb5kdc[12163](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 09:02:37 dist-test-slave-mtdr krb5kdc[12163](info): set up 2 sockets
Jun 19 09:02:37 dist-test-slave-mtdr krb5kdc[12163](info): commencing operation
krb5kdc: starting...
W20260619 09:02:38.798481 392 mini_kdc.cc:121] Time spent starting KDC: real 0.941s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859658134510-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859658134510-392-0/krb5kdc/test-user.keytab.
Jun 19 09:02:38 dist-test-slave-mtdr krb5kdc[12163](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859758, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T09:02:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T09:02:38Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859658134510-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859658134510-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859658134510-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859658134510-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859658134510-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859658134510-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
I20260619 09:02:38.949638 392 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859658134510-392-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task4ez5NB/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859658134510-392-0/minicluster-data/postgres -l logfile start
2026-06-19 09:02:41.269 UTC [12193] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-19 09:02:41.269 UTC [12193] LOG: listening on IPv4 address "127.0.98.20", port 42583
2026-06-19 09:02:41.273 UTC [12193] LOG: listening on Unix socket "/tmp/.s.PGSQL.42583"
2026-06-19 09:02:41.291 UTC [12196] LOG: database system was shut down at 2026-06-19 09:02:39 UTC
2026-06-19 09:02:41.295 UTC [12193] LOG: database system is ready to accept connections
I20260619 09:02:42.167337 392 mini_postgres.cc:96] Postgres bound to 42583
2026-06-19 09:02:42.173 UTC [12204] FATAL: database "slave" does not exist
127.0.98.20:42583 - accepting connections
I20260619 09:02:42.174295 392 mini_ranger.cc:161] Starting Ranger...
I20260619 09:02:42.190701 392 mini_ranger.cc:84] Created miniranger Postgres user
I20260619 09:02:42.641148 392 mini_ranger.cc:87] Created ranger Postgres database
I20260619 09:02:42.641216 392 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859658134510-392-0/minicluster-data/ranger-admin
2026-06-19 09:02:42,895 [I] DB FLAVOR :POSTGRES
2026-06-19 09:02:42,896 [I] --------- Verifying Ranger DB connection ---------
2026-06-19 09:02:42,896 [I] Checking connection..
2026-06-19 09:02:42,896 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4ez5NB/build/release/bin/postgresql.jar:/tmp/dist-test-task4ez5NB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:42583/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-19 09:02:43,143 [I] Checking connection passed.
2026-06-19 09:02:43,143 [I] --------- Verifying version history table ---------
2026-06-19 09:02:43,143 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4ez5NB/build/release/bin/postgresql.jar:/tmp/dist-test-task4ez5NB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:42583/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 09:02:43,391 [I] Table x_db_version_h does not exist in database ranger
2026-06-19 09:02:43,392 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4ez5NB/build/release/bin/postgresql.jar:/tmp/dist-test-task4ez5NB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:42583/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 09:02:43,634 [I] Table x_db_version_h does not exist in database ranger
2026-06-19 09:02:43,635 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-19 09:02:43,635 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4ez5NB/build/release/bin/postgresql.jar:/tmp/dist-test-task4ez5NB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:42583/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task4ez5NB/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-19 09:02:43.886 UTC [12268] WARNING: there is no transaction in progress
2026-06-19 09:02:43,891 [I] create_dbversion_catalog.sql file imported successfully
2026-06-19 09:02:43,891 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4ez5NB/build/release/bin/postgresql.jar:/tmp/dist-test-task4ez5NB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:42583/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 09:02:44,131 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-19 09:02:44,131 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-19 09:02:44,131 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4ez5NB/build/release/bin/postgresql.jar:/tmp/dist-test-task4ez5NB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:42583/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-19 09:02:44,364 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4ez5NB/build/release/bin/postgresql.jar:/tmp/dist-test-task4ez5NB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:42583/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-19 09:02:44,608 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4ez5NB/build/release/bin/postgresql.jar:/tmp/dist-test-task4ez5NB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:42583/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-mtdr.c.gcp-upstream.internal','N') ;"
2026-06-19 09:02:44,821 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4ez5NB/build/release/bin/postgresql.jar:/tmp/dist-test-task4ez5NB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:42583/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-19 09:02:45,042 [I] Table x_portal_user does not exist in database ranger
2026-06-19 09:02:45,042 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4ez5NB/build/release/bin/postgresql.jar:/tmp/dist-test-task4ez5NB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:42583/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-19 09:02:45,289 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-19 09:02:45,290 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4ez5NB/build/release/bin/postgresql.jar:/tmp/dist-test-task4ez5NB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:42583/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-19 09:02:45,505 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4ez5NB/build/release/bin/postgresql.jar:/tmp/dist-test-task4ez5NB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:42583/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-19 09:02:45,741 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-19 09:02:45,742 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4ez5NB/build/release/bin/postgresql.jar:/tmp/dist-test-task4ez5NB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:42583/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task4ez5NB/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-19 09:02:46.655 UTC [12376] WARNING: there is no transaction in progress
2026-06-19 09:02:46.672 UTC [12376] WARNING: there is no transaction in progress
2026-06-19 09:02:46.687 UTC [12376] WARNING: there is no transaction in progress
2026-06-19 09:02:46.702 UTC [12376] WARNING: there is no transaction in progress
2026-06-19 09:02:46.718 UTC [12376] WARNING: there is no transaction in progress
2026-06-19 09:02:46.735 UTC [12376] WARNING: there is no transaction in progress
2026-06-19 09:02:46.789 UTC [12376] WARNING: there is no transaction in progress
2026-06-19 09:02:46.795 UTC [12376] WARNING: there is no transaction in progress
2026-06-19 09:02:46.809 UTC [12376] WARNING: there is no transaction in progress
2026-06-19 09:02:46.819 UTC [12376] WARNING: there is no transaction in progress
2026-06-19 09:02:46.829 UTC [12376] WARNING: there is no transaction in progress
2026-06-19 09:02:46.843 UTC [12376] WARNING: there is no transaction in progress
2026-06-19 09:02:46.853 UTC [12376] WARNING: there is no transaction in progress
2026-06-19 09:02:46.860 UTC [12376] WARNING: there is no transaction in progress
2026-06-19 09:02:46.867 UTC [12376] WARNING: there is no transaction in progress
2026-06-19 09:02:46.874 UTC [12376] WARNING: there is no transaction in progress
2026-06-19 09:02:47,515 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-19 09:02:47,515 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4ez5NB/build/release/bin/postgresql.jar:/tmp/dist-test-task4ez5NB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:42583/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-19 09:02:47,757 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-19 09:02:47,758 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4ez5NB/build/release/bin/postgresql.jar:/tmp/dist-test-task4ez5NB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:42583/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-19 09:02:47,997 [I] Table x_portal_user already exists in database 'ranger'
2026-06-19 09:02:47,998 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4ez5NB/build/release/bin/postgresql.jar:/tmp/dist-test-task4ez5NB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:42583/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-19 09:02:48,222 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-19 09:02:48,222 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4ez5NB/build/release/bin/postgresql.jar:/tmp/dist-test-task4ez5NB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:42583/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-19 09:02:48,462 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4ez5NB/build/release/bin/postgresql.jar:/tmp/dist-test-task4ez5NB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:42583/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-19 09:02:48,711 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4ez5NB/build/release/bin/postgresql.jar:/tmp/dist-test-task4ez5NB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:42583/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-mtdr.c.gcp-upstream.internal';"
2026-06-19 09:02:48,953 [I] CORE_DB_SCHEMA import status has been updated
2026-06-19 09:02:48,954 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4ez5NB/build/release/bin/postgresql.jar:/tmp/dist-test-task4ez5NB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:42583/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-19 09:02:49,198 [I] DB_PATCHES have already been applied
I20260619 09:02:49.203929 392 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859658134510-392-0/minicluster-data/ranger-admin:/tmp/dist-test-task4ez5NB/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task4ez5NB/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task4ez5NB/build/release/bin/postgresql.jar:/tmp/dist-test-task4ez5NB/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260619 09:02:49.203999 392 mini_ranger.cc:193] Using host: 127.0.98.20
I20260619 09:02:49.206745 392 mini_ranger.cc:227] Ranger admin URL: http://127.0.98.20:58445
Jun 19, 2026 9:02:49 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 19, 2026 9:02:49 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 19, 2026 9:02:49 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859658134510-392-0/minicluster-data/ranger-admin/ews/webapp
Jun 19, 2026 9:02:49 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859658134510-392-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 19, 2026 9:02:49 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859658134510-392-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 19, 2026 9:02:50 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859658134510-392-0/minicluster-data/ranger-admin/ews/webapp].
Jun 19, 2026 9:02:50 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260619 09:02:50.200292 392 mini_ranger.cc:160] Time spent starting Ranger: real 8.026s user 0.000s sys 0.008s
Jun 19, 2026 9:03:09 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 19, 2026 9:03:09 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.XKeyREST
Jun 19, 2026 9:03:09 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 19, 2026 9:03:09 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 19, 2026 9:03:09 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 19, 2026 9:03:09 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 19, 2026 9:03:09 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 19, 2026 9:03:09 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 19, 2026 9:03:09 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 19, 2026 9:03:09 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 19, 2026 9:03:09 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 19, 2026 9:03:09 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 19, 2026 9:03:09 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 19, 2026 9:03:09 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 19, 2026 9:03:09 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 19, 2026 9:03:09 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 19, 2026 9:03:09 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 19, 2026 9:03:10 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260619 09:03:10.623378 392 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859658134510-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859658134510-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.98.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859658134510-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859658134510-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 19 09:03:10 dist-test-slave-mtdr krb5kdc[12163](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859790, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260619 09:03:10.697399 392 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260619 09:03:10.705330 392 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260619 09:03:11.508303 392 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260619 09:03:11.508383 392 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859658134510-392-0/minicluster-data/ranger-kms
I20260619 09:03:11.508399 392 mini_ranger_kms.cc:227] Using postgres at 127.0.98.20:42583
2026-06-19 09:03:11,650 [I] DB FLAVOR :POSTGRES
2026-06-19 09:03:11,650 [I] --------- Verifying Ranger DB connection ---------
2026-06-19 09:03:11,650 [I] Checking connection
2026-06-19 09:03:11,650 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4ez5NB/build/release/bin/postgresql.jar:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859658134510-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:42583/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-19 09:03:11,876 [I] connection success
2026-06-19 09:03:11,876 [I] --------- Verifying Ranger DB tables ---------
2026-06-19 09:03:11,876 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-19 09:03:11,876 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4ez5NB/build/release/bin/postgresql.jar:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859658134510-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:42583/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-19 09:03:12,105 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-19 09:03:12,105 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-19 09:03:12,105 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-19 09:03:12,105 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task4ez5NB/build/release/bin/postgresql.jar:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859658134510-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:42583/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859658134510-392-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-19 09:03:12,355 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260619 09:03:12.828910 392 mini_ranger_kms.cc:310] Created kms service
I20260619 09:03:13.032823 392 mini_ranger_kms.cc:326] Created kudu user
I20260619 09:03:13.073939 392 mini_ranger_kms.cc:343] Created rangerkms user
Jun 19 09:03:13 dist-test-slave-mtdr krb5kdc[12163](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859790, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260619 09:03:13.456491 392 mini_ranger_kms.cc:384] Added ranger policy
I20260619 09:03:13.456602 392 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859658134510-392-0/minicluster-data/ranger-kms:/tmp/dist-test-task4ez5NB/build/release/bin/postgresql.jar:/tmp/dist-test-task4ez5NB/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task4ez5NB/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260619 09:03:13.456620 392 mini_ranger_kms.cc:240] Using host: 127.0.98.20
I20260619 09:03:13.459841 392 mini_ranger_kms.cc:276] Ranger KMS PID: 12556
I20260619 09:03:13.459920 392 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.98.20:48805
Jun 19, 2026 9:03:14 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 19, 2026 9:03:14 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 19, 2026 9:03:14 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859658134510-392-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 19, 2026 9:03:14 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859658134510-392-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 19, 2026 9:03:14 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859658134510-392-0/minicluster-data/ranger-kms/ews/webapp].
Jun 19, 2026 9:03:14 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260619 09:03:15.506505 392 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 4.809s user 0.002s sys 0.122s
I20260619 09:03:15.506608 392 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260619 09:03:15.506652 392 mini_ranger_kms.cc:401] 127.0.98.20:48805/kms/v1/keys
Jun 19 09:03:15 dist-test-slave-mtdr krb5kdc[12163](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781859795, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Jun 19 09:03:15 dist-test-slave-mtdr krb5kdc[12163](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859795, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Jun 19 09:03:16 dist-test-slave-mtdr krb5kdc[12163](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859795, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Jun 19 09:03:18 dist-test-slave-mtdr krb5kdc[12163](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859798, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859658134510-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859658134510-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 09:03:18.421811 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ez5NB/build/release/bin/kudu
/tmp/dist-test-task4ez5NB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859658134510-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859658134510-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859658134510-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859658134510-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44911
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859658134510-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44911
--ranger_config_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859658134510-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:48805/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859658134510-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859658134510-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859658134510-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 09:03:18.520326 12607 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 09:03:18.520510 12607 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 09:03:18.520530 12607 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 09:03:18.521651 12607 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 09:03:18.521687 12607 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 09:03:18.521700 12607 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 09:03:18.521711 12607 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 09:03:18.521721 12607 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 09:03:18.523273 12607 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859658134510-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859658134510-392-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.98.20:48805/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44911
--ranger_config_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859658134510-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859658134510-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44911
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859658134510-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12607
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859658134510-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 09:03:18.523443 12607 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 09:03:18.523594 12607 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 09:03:18.527913 12615 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:03:18.528039 12607 server_base.cc:1034] running on GCE node
W20260619 09:03:18.528067 12612 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:03:18.527918 12613 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:03:18.528388 12607 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 09:03:18.528642 12607 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 09:03:18.529860 12607 hybrid_clock.cc:648] HybridClock initialized: now 1781859798529847 us; error 25 us; skew 500 ppm
Jun 19 09:03:18 dist-test-slave-mtdr krb5kdc[12163](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859798, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 09:03:18.535740 12607 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 09:03:18.536164 12607 webserver.cc:460] Webserver started at http://127.0.98.62:42341/ using document root <none> and password file <none>
I20260619 09:03:18.536361 12607 fs_manager.cc:359] Metadata directory not provided
I20260619 09:03:18.536401 12607 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 09:03:18.536516 12607 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 09:03:18 dist-test-slave-mtdr krb5kdc[12163](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859798, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260619 09:03:18.589294 12607 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859658134510-392-0/minicluster-data/master-0/data/instance:
uuid: "0b84830195484d8e96ffc8d8eb5ce0de"
format_stamp: "Formatted at 2026-06-19 09:03:18 on dist-test-slave-mtdr"
server_key: "a612ae6a3833955079c40bf4ecfff37d"
server_key_iv: "85028f6bfc716f8676554b52d9598d00"
server_key_version: "kuduclusterkey@0"
I20260619 09:03:18.589730 12607 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859658134510-392-0/minicluster-data/master-0/wal/instance:
uuid: "0b84830195484d8e96ffc8d8eb5ce0de"
format_stamp: "Formatted at 2026-06-19 09:03:18 on dist-test-slave-mtdr"
server_key: "a612ae6a3833955079c40bf4ecfff37d"
server_key_iv: "85028f6bfc716f8676554b52d9598d00"
server_key_version: "kuduclusterkey@0"
I20260619 09:03:18.591565 12607 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260619 09:03:18.607010 12622 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 09:03:18.607203 12607 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260619 09:03:18.607298 12607 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859658134510-392-0/minicluster-data/master-0/data,/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859658134510-392-0/minicluster-data/master-0/wal
uuid: "0b84830195484d8e96ffc8d8eb5ce0de"
format_stamp: "Formatted at 2026-06-19 09:03:18 on dist-test-slave-mtdr"
server_key: "a612ae6a3833955079c40bf4ecfff37d"
server_key_iv: "85028f6bfc716f8676554b52d9598d00"
server_key_version: "kuduclusterkey@0"
I20260619 09:03:18.607406 12607 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859658134510-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859658134510-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859658134510-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 09:03:18.622529 12607 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 09:03:18.624413 12607 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 09:03:18.624549 12607 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 09:03:18.628614 12607 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44911
I20260619 09:03:18.628657 12674 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44911 every 8 connection(s)
I20260619 09:03:18.628990 12607 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859658134510-392-0/minicluster-data/master-0/data/info.pb
I20260619 09:03:18.629757 12675 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 09:03:18.632820 12675 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b84830195484d8e96ffc8d8eb5ce0de: Bootstrap starting.
I20260619 09:03:18.633510 12675 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0b84830195484d8e96ffc8d8eb5ce0de: Neither blocks nor log segments found. Creating new log.
I20260619 09:03:18.633821 12675 log.cc:824] T 00000000000000000000000000000000 P 0b84830195484d8e96ffc8d8eb5ce0de: Log is configured to *not* fsync() on all Append() calls
I20260619 09:03:18.634598 12675 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b84830195484d8e96ffc8d8eb5ce0de: No bootstrap required, opened a new log
I20260619 09:03:18.637444 12675 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0b84830195484d8e96ffc8d8eb5ce0de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b84830195484d8e96ffc8d8eb5ce0de" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44911 } }
I20260619 09:03:18.637633 12675 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0b84830195484d8e96ffc8d8eb5ce0de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 09:03:18.637665 12675 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0b84830195484d8e96ffc8d8eb5ce0de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b84830195484d8e96ffc8d8eb5ce0de, State: Initialized, Role: FOLLOWER
I20260619 09:03:18.637848 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ez5NB/build/release/bin/kudu as pid 12607
I20260619 09:03:18.637848 12675 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0b84830195484d8e96ffc8d8eb5ce0de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b84830195484d8e96ffc8d8eb5ce0de" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44911 } }
I20260619 09:03:18.637915 12675 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0b84830195484d8e96ffc8d8eb5ce0de [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 09:03:18.637954 12675 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0b84830195484d8e96ffc8d8eb5ce0de [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 09:03:18.637974 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859658134510-392-0/minicluster-data/master-0/wal/instance
I20260619 09:03:18.638031 12675 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0b84830195484d8e96ffc8d8eb5ce0de [term 0 FOLLOWER]: Advancing to term 1
I20260619 09:03:18.638172 392 external_mini_cluster.cc:1428] Setting key 8c3884401219bf7a53ee21dec6d5d957
I20260619 09:03:18.638723 12675 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0b84830195484d8e96ffc8d8eb5ce0de [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b84830195484d8e96ffc8d8eb5ce0de" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44911 } }
I20260619 09:03:18.638921 12675 leader_election.cc:302] T 00000000000000000000000000000000 P 0b84830195484d8e96ffc8d8eb5ce0de [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0b84830195484d8e96ffc8d8eb5ce0de; no voters:
I20260619 09:03:18.639108 12675 leader_election.cc:288] T 00000000000000000000000000000000 P 0b84830195484d8e96ffc8d8eb5ce0de [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 09:03:18.639145 12680 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0b84830195484d8e96ffc8d8eb5ce0de [term 1 FOLLOWER]: Leader election won for term 1
I20260619 09:03:18.639384 12680 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0b84830195484d8e96ffc8d8eb5ce0de [term 1 LEADER]: Becoming Leader. State: Replica: 0b84830195484d8e96ffc8d8eb5ce0de, State: Running, Role: LEADER
I20260619 09:03:18.639432 12675 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0b84830195484d8e96ffc8d8eb5ce0de [sys.catalog]: configured and running, proceeding with master startup.
I20260619 09:03:18.639544 12680 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0b84830195484d8e96ffc8d8eb5ce0de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b84830195484d8e96ffc8d8eb5ce0de" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44911 } }
I20260619 09:03:18.639966 12680 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b84830195484d8e96ffc8d8eb5ce0de [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0b84830195484d8e96ffc8d8eb5ce0de. Latest consensus state: current_term: 1 leader_uuid: "0b84830195484d8e96ffc8d8eb5ce0de" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b84830195484d8e96ffc8d8eb5ce0de" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44911 } } }
I20260619 09:03:18.640100 12680 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b84830195484d8e96ffc8d8eb5ce0de [sys.catalog]: This master's current role is: LEADER
I20260619 09:03:18.640333 12689 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 09:03:18.640848 12689 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 09:03:18.641037 12681 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b84830195484d8e96ffc8d8eb5ce0de [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0b84830195484d8e96ffc8d8eb5ce0de" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b84830195484d8e96ffc8d8eb5ce0de" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44911 } } }
I20260619 09:03:18.641124 12681 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b84830195484d8e96ffc8d8eb5ce0de [sys.catalog]: This master's current role is: LEADER
I20260619 09:03:18.641409 12675 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859658134510-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
Jun 19 09:03:18 dist-test-slave-mtdr krb5kdc[12163](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859798, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 09:03:18.651929 12679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:03:18.638643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56848 (local address 127.0.98.62:44911)
0619 09:03:18.638835 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0619 09:03:18.638846 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 09:03:18.638878 (+ 32us) server_negotiation.cc:407] Connection header received
0619 09:03:18.638991 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:03:18.638999 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:03:18.639058 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:03:18.640016 (+ 958us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:03:18.640116 (+ 100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:03:18.641529 (+ 1413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:03:18.643317 (+ 1788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:03:18.644715 (+ 1398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:03:18.645220 (+ 505us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:03:18.647721 (+ 2501us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:03:18.647727 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:03:18.647727 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:03:18.647738 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:03:18.649846 (+ 2108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:03:18.650258 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:03:18.650261 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:03:18.650262 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:03:18.650368 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:03:18.650574 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:03:18.650575 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:03:18.650575 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:03:18.650930 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:03:18.651475 (+ 545us) server_negotiation.cc:1035] Waiting for connection context
0619 09:03:18.651553 (+ 78us) server_negotiation.cc:299] Negotiation successful
0619 09:03:18.651685 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":69,"threads_started":1}
I20260619 09:03:18.652983 12689 catalog_manager.cc:1347] Generated new cluster ID: a54cc6f58a6841d2812e2d89720fa371
I20260619 09:03:18.653033 12689 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 09:03:18.664619 12689 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 09:03:18.665334 12689 catalog_manager.cc:1504] Loading token signing keys...
I20260619 09:03:18.671907 12689 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0b84830195484d8e96ffc8d8eb5ce0de: Generated new TSK 0
I20260619 09:03:18.672088 12689 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859658134510-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859658134510-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 09:03:19.299552 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ez5NB/build/release/bin/kudu
/tmp/dist-test-task4ez5NB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859658134510-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859658134510-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859658134510-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859658134510-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:48805/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44911
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859658134510-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859658134510-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859658134510-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859658134510-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 09:03:19.384258 12720 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 09:03:19.384449 12720 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 09:03:19.384471 12720 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 09:03:19.385682 12720 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 09:03:19.385726 12720 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 09:03:19.385759 12720 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 09:03:19.387269 12720 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859658134510-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859658134510-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.98.20:48805/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859658134510-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859658134510-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44911
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12720
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859658134510-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 09:03:19.387490 12720 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 09:03:19.387717 12720 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 09:03:19.392386 12726 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:03:19.392391 12728 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:03:19.392678 12725 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:03:19.393051 12720 server_base.cc:1034] running on GCE node
I20260619 09:03:19.393289 12720 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 09:03:19.393579 12720 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 09:03:19.394752 12720 hybrid_clock.cc:648] HybridClock initialized: now 1781859799394730 us; error 33 us; skew 500 ppm
Jun 19 09:03:19 dist-test-slave-mtdr krb5kdc[12163](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859799, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 09:03:19.397851 12720 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 09:03:19.398427 12720 webserver.cc:460] Webserver started at http://127.0.98.1:41875/ using document root <none> and password file <none>
I20260619 09:03:19.398702 12720 fs_manager.cc:359] Metadata directory not provided
I20260619 09:03:19.398766 12720 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 09:03:19.398917 12720 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 09:03:19 dist-test-slave-mtdr krb5kdc[12163](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859799, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260619 09:03:19.414407 12720 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859658134510-392-0/minicluster-data/ts-0/data/instance:
uuid: "412d20e190f64b59b233e4b7da26de6b"
format_stamp: "Formatted at 2026-06-19 09:03:19 on dist-test-slave-mtdr"
server_key: "96998d4b10e5983a4be1d57e99e8081e"
server_key_iv: "1bd7e0d82dac79e05ab93e4bd37ed610"
server_key_version: "kuduclusterkey@0"
I20260619 09:03:19.414798 12720 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859658134510-392-0/minicluster-data/ts-0/wal/instance:
uuid: "412d20e190f64b59b233e4b7da26de6b"
format_stamp: "Formatted at 2026-06-19 09:03:19 on dist-test-slave-mtdr"
server_key: "96998d4b10e5983a4be1d57e99e8081e"
server_key_iv: "1bd7e0d82dac79e05ab93e4bd37ed610"
server_key_version: "kuduclusterkey@0"
I20260619 09:03:19.416633 12720 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.001s
I20260619 09:03:19.428185 12735 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 09:03:19.428334 12720 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 09:03:19.428416 12720 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859658134510-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859658134510-392-0/minicluster-data/ts-0/wal
uuid: "412d20e190f64b59b233e4b7da26de6b"
format_stamp: "Formatted at 2026-06-19 09:03:19 on dist-test-slave-mtdr"
server_key: "96998d4b10e5983a4be1d57e99e8081e"
server_key_iv: "1bd7e0d82dac79e05ab93e4bd37ed610"
server_key_version: "kuduclusterkey@0"
I20260619 09:03:19.428505 12720 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859658134510-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859658134510-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859658134510-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 09:03:19.446476 12720 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 09:03:19.448768 12720 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 09:03:19.448968 12720 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 09:03:19.449287 12720 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 09:03:19.449702 12720 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 09:03:19.449786 12720 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 09:03:19.449887 12720 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 09:03:19.449959 12720 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 09:03:19.456516 12720 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40147
I20260619 09:03:19.456956 12720 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859658134510-392-0/minicluster-data/ts-0/data/info.pb
I20260619 09:03:19.457161 12849 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40147 every 8 connection(s)
Jun 19 09:03:19 dist-test-slave-mtdr krb5kdc[12163](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859799, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 09:03:19.464501 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ez5NB/build/release/bin/kudu as pid 12720
I20260619 09:03:19.464597 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859658134510-392-0/minicluster-data/ts-0/wal/instance
I20260619 09:03:19.464792 392 external_mini_cluster.cc:1428] Setting key bcb3a7613acfb21061cbff54b3c22234
I20260619 09:03:19.468919 12853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:03:19.457953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44911 (local address 127.0.98.1:51713)
0619 09:03:19.458137 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0619 09:03:19.458182 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 09:03:19.458793 (+ 611us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:03:19.459197 (+ 404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:03:19.459202 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:03:19.459433 (+ 231us) client_negotiation.cc:189] Negotiated authn=SASL
0619 09:03:19.459703 (+ 270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:03:19.459710 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:03:19.460665 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:03:19.460667 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:03:19.461617 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:03:19.461620 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:03:19.462135 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:03:19.462137 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:03:19.462270 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:03:19.462873 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 09:03:19.462883 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 09:03:19.464218 (+ 1335us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 09:03:19.467283 (+ 3065us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:03:19.467290 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:03:19.467293 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:03:19.467590 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:03:19.467805 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:03:19.467806 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:03:19.467806 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:03:19.467941 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:03:19.468208 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 09:03:19.468213 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 09:03:19.468391 (+ 178us) client_negotiation.cc:769] Sending connection context
0619 09:03:19.468489 (+ 98us) client_negotiation.cc:240] Negotiation successful
0619 09:03:19.468657 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":109,"threads_started":1}
I20260619 09:03:19.469601 12852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:03:19.457966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51713 (local address 127.0.98.62:44911)
0619 09:03:19.458100 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0619 09:03:19.458102 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:03:19.458952 (+ 850us) server_negotiation.cc:407] Connection header received
0619 09:03:19.458983 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:03:19.458985 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:03:19.458999 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:03:19.459071 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:03:19.459815 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:03:19.460565 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:03:19.461702 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:03:19.461989 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:03:19.462058 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:03:19.464304 (+ 2246us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:03:19.464306 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:03:19.464306 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:03:19.464314 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:03:19.467197 (+ 2883us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:03:19.467672 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:03:19.467673 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:03:19.467674 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:03:19.467741 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:03:19.468002 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:03:19.468003 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:03:19.468003 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:03:19.468133 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:03:19.468186 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0619 09:03:19.469419 (+ 1233us) server_negotiation.cc:299] Negotiation successful
0619 09:03:19.469502 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105,"thread_start_us":74,"threads_started":1}
I20260619 09:03:19.470332 12850 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44911
I20260619 09:03:19.470450 12850 heartbeater.cc:464] Registering TS with master...
I20260619 09:03:19.470669 12850 heartbeater.cc:511] Master 127.0.98.62:44911 requested a full tablet report, sending...
I20260619 09:03:19.471241 12639 ts_manager.cc:194] Registered new tserver with Master: 412d20e190f64b59b233e4b7da26de6b (127.0.98.1:40147)
I20260619 09:03:19.471841 12639 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:51713
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859658134510-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859658134510-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 09:03:19.491034 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ez5NB/build/release/bin/kudu
/tmp/dist-test-task4ez5NB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859658134510-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859658134510-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859658134510-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859658134510-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:48805/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44911
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859658134510-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859658134510-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859658134510-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859658134510-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 09:03:19.597877 12857 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 09:03:19.598153 12857 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 09:03:19.598186 12857 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 09:03:19.599942 12857 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 09:03:19.600006 12857 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 09:03:19.600045 12857 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 09:03:19.602207 12857 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859658134510-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859658134510-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.98.20:48805/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859658134510-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859658134510-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44911
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12857
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859658134510-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 09:03:19.602454 12857 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 09:03:19.602802 12857 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 09:03:19.608906 12864 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:03:19.608916 12867 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:03:19.609786 12865 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:03:19.610028 12857 server_base.cc:1034] running on GCE node
I20260619 09:03:19.610235 12857 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 09:03:19.610509 12857 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 09:03:19.611690 12857 hybrid_clock.cc:648] HybridClock initialized: now 1781859799611666 us; error 35 us; skew 500 ppm
Jun 19 09:03:19 dist-test-slave-mtdr krb5kdc[12163](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859799, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 09:03:19.614866 12857 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 09:03:19.615370 12857 webserver.cc:460] Webserver started at http://127.0.98.2:42285/ using document root <none> and password file <none>
I20260619 09:03:19.615634 12857 fs_manager.cc:359] Metadata directory not provided
I20260619 09:03:19.615706 12857 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 09:03:19.615877 12857 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 09:03:19 dist-test-slave-mtdr krb5kdc[12163](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859799, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Jun 19 09:03:19 dist-test-slave-mtdr krb5kdc[12163](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781859799, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 09:03:19.636416 12857 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859658134510-392-0/minicluster-data/ts-1/data/instance:
uuid: "5712911e376c4b78b0d6c49a9495c911"
format_stamp: "Formatted at 2026-06-19 09:03:19 on dist-test-slave-mtdr"
server_key: "52f5d74bf3b84e8b42acf09563150d84"
server_key_iv: "7ebb1b22e43b108cf19129790b20c16a"
server_key_version: "kuduclusterkey@0"
I20260619 09:03:19.636871 12857 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859658134510-392-0/minicluster-data/ts-1/wal/instance:
uuid: "5712911e376c4b78b0d6c49a9495c911"
format_stamp: "Formatted at 2026-06-19 09:03:19 on dist-test-slave-mtdr"
server_key: "52f5d74bf3b84e8b42acf09563150d84"
server_key_iv: "7ebb1b22e43b108cf19129790b20c16a"
server_key_version: "kuduclusterkey@0"
I20260619 09:03:19.638808 12857 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260619 09:03:19.678947 12874 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 09:03:19.679173 12857 fs_manager.cc:725] Time spent opening block manager: real 0.010s user 0.001s sys 0.000s
I20260619 09:03:19.679284 12857 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859658134510-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859658134510-392-0/minicluster-data/ts-1/wal
uuid: "5712911e376c4b78b0d6c49a9495c911"
format_stamp: "Formatted at 2026-06-19 09:03:19 on dist-test-slave-mtdr"
server_key: "52f5d74bf3b84e8b42acf09563150d84"
server_key_iv: "7ebb1b22e43b108cf19129790b20c16a"
server_key_version: "kuduclusterkey@0"
I20260619 09:03:19.679399 12857 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859658134510-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859658134510-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859658134510-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 09:03:19.698539 12857 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 09:03:19.700891 12857 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 09:03:19.701099 12857 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 09:03:19.701427 12857 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 09:03:19.701937 12857 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 09:03:19.702018 12857 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 09:03:19.702083 12857 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 09:03:19.702135 12857 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 09:03:19.708914 12857 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33713
I20260619 09:03:19.709396 12857 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859658134510-392-0/minicluster-data/ts-1/data/info.pb
I20260619 09:03:19.710042 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ez5NB/build/release/bin/kudu as pid 12857
I20260619 09:03:19.710155 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859658134510-392-0/minicluster-data/ts-1/wal/instance
I20260619 09:03:19.710359 392 external_mini_cluster.cc:1428] Setting key 78dffd61d99264a16886dabf493f27ae
I20260619 09:03:19.715807 12988 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33713 every 8 connection(s)
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Jun 19 09:03:19 dist-test-slave-mtdr krb5kdc[12163](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859799, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 09:03:19.734745 12852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:03:19.716752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55703 (local address 127.0.98.62:44911)
0619 09:03:19.716798 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0619 09:03:19.716800 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:03:19.720911 (+ 4111us) server_negotiation.cc:407] Connection header received
0619 09:03:19.721749 (+ 838us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:03:19.721750 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:03:19.721764 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:03:19.721795 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:03:19.722691 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:03:19.723514 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:03:19.726618 (+ 3104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:03:19.726935 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:03:19.726983 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:03:19.729465 (+ 2482us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:03:19.729466 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:03:19.729467 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:03:19.729473 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:03:19.731012 (+ 1539us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:03:19.733617 (+ 2605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:03:19.733618 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:03:19.733619 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:03:19.733689 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:03:19.733975 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:03:19.733976 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:03:19.733976 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:03:19.734113 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:03:19.734170 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0619 09:03:19.734545 (+ 375us) server_negotiation.cc:299] Negotiation successful
0619 09:03:19.734641 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260619 09:03:19.735270 12992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:03:19.716470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44911 (local address 127.0.98.2:55703)
0619 09:03:19.720738 (+ 4268us) negotiation.cc:107] Waiting for socket to connect
0619 09:03:19.720791 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0619 09:03:19.721489 (+ 698us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:03:19.721989 (+ 500us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:03:19.721994 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:03:19.722234 (+ 240us) client_negotiation.cc:189] Negotiated authn=SASL
0619 09:03:19.722582 (+ 348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:03:19.722589 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:03:19.725238 (+ 2649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:03:19.725239 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:03:19.726521 (+ 1282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:03:19.726523 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:03:19.727087 (+ 564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:03:19.727088 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:03:19.727237 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:03:19.727790 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 09:03:19.727800 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 09:03:19.729363 (+ 1563us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 09:03:19.731091 (+ 1728us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:03:19.731098 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:03:19.731100 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:03:19.731372 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:03:19.733762 (+ 2390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:03:19.733763 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:03:19.733764 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:03:19.733901 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:03:19.734213 (+ 312us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 09:03:19.734219 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 09:03:19.734391 (+ 172us) client_negotiation.cc:769] Sending connection context
0619 09:03:19.734866 (+ 475us) client_negotiation.cc:240] Negotiation successful
0619 09:03:19.735027 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4218,"thread_start_us":82,"threads_started":1}
I20260619 09:03:19.735737 12989 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44911
I20260619 09:03:19.735836 12989 heartbeater.cc:464] Registering TS with master...
I20260619 09:03:19.736032 12989 heartbeater.cc:511] Master 127.0.98.62:44911 requested a full tablet report, sending...
I20260619 09:03:19.736460 12639 ts_manager.cc:194] Registered new tserver with Master: 5712911e376c4b78b0d6c49a9495c911 (127.0.98.2:33713)
I20260619 09:03:19.736961 12639 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:55703
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859658134510-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859658134510-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 09:03:19.751793 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ez5NB/build/release/bin/kudu
/tmp/dist-test-task4ez5NB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859658134510-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859658134510-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859658134510-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859658134510-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:48805/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44911
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859658134510-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859658134510-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859658134510-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859658134510-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 09:03:19.873915 12994 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 09:03:19.874220 12994 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 09:03:19.874264 12994 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 09:03:19.876067 12994 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 09:03:19.876137 12994 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 09:03:19.876194 12994 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 09:03:19.878584 12994 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859658134510-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859658134510-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.98.20:48805/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859658134510-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859658134510-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44911
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12994
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859658134510-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 09:03:19.878866 12994 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 09:03:19.879159 12994 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 09:03:19.885782 13000 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:03:19.885794 13002 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:03:19.886315 12999 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:03:19.886965 12994 server_base.cc:1034] running on GCE node
I20260619 09:03:19.887180 12994 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 09:03:19.887431 12994 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 09:03:19.890722 12994 hybrid_clock.cc:648] HybridClock initialized: now 1781859799890694 us; error 41 us; skew 500 ppm
Jun 19 09:03:19 dist-test-slave-mtdr krb5kdc[12163](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859799, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 09:03:19.893455 12994 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 09:03:19.894001 12994 webserver.cc:460] Webserver started at http://127.0.98.3:43683/ using document root <none> and password file <none>
I20260619 09:03:19.894254 12994 fs_manager.cc:359] Metadata directory not provided
I20260619 09:03:19.894318 12994 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 09:03:19.894446 12994 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 09:03:19 dist-test-slave-mtdr krb5kdc[12163](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859799, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260619 09:03:19.907984 12994 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859658134510-392-0/minicluster-data/ts-2/data/instance:
uuid: "8bf3e72f7bc44976a8b60b1e5473c061"
format_stamp: "Formatted at 2026-06-19 09:03:19 on dist-test-slave-mtdr"
server_key: "d126301ff7d8eeaa8f5ea2e0a9bab6f4"
server_key_iv: "bfbc19ae73becc1e23af4f4ab3fcb2a8"
server_key_version: "kuduclusterkey@0"
I20260619 09:03:19.908428 12994 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859658134510-392-0/minicluster-data/ts-2/wal/instance:
uuid: "8bf3e72f7bc44976a8b60b1e5473c061"
format_stamp: "Formatted at 2026-06-19 09:03:19 on dist-test-slave-mtdr"
server_key: "d126301ff7d8eeaa8f5ea2e0a9bab6f4"
server_key_iv: "bfbc19ae73becc1e23af4f4ab3fcb2a8"
server_key_version: "kuduclusterkey@0"
I20260619 09:03:19.910058 12994 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260619 09:03:19.923928 13009 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 09:03:19.924175 12994 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 09:03:19.924281 12994 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859658134510-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859658134510-392-0/minicluster-data/ts-2/wal
uuid: "8bf3e72f7bc44976a8b60b1e5473c061"
format_stamp: "Formatted at 2026-06-19 09:03:19 on dist-test-slave-mtdr"
server_key: "d126301ff7d8eeaa8f5ea2e0a9bab6f4"
server_key_iv: "bfbc19ae73becc1e23af4f4ab3fcb2a8"
server_key_version: "kuduclusterkey@0"
I20260619 09:03:19.924386 12994 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859658134510-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859658134510-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859658134510-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 09:03:19.940201 12994 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 09:03:19.942073 12994 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 09:03:19.942190 12994 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 09:03:19.942404 12994 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 09:03:19.942745 12994 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 09:03:19.942783 12994 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 09:03:19.942804 12994 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 09:03:19.942847 12994 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 09:03:19.949733 12994 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39527
I20260619 09:03:19.950131 12994 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859658134510-392-0/minicluster-data/ts-2/data/info.pb
I20260619 09:03:19.951529 13122 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39527 every 8 connection(s)
Jun 19 09:03:19 dist-test-slave-mtdr krb5kdc[12163](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859799, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 09:03:19.958671 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ez5NB/build/release/bin/kudu as pid 12994
I20260619 09:03:19.958778 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859658134510-392-0/minicluster-data/ts-2/wal/instance
I20260619 09:03:19.959003 392 external_mini_cluster.cc:1428] Setting key fb0c1a35ddf2c480a57488ca83909cde
I20260619 09:03:19.961715 12852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:03:19.952097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51531 (local address 127.0.98.62:44911)
0619 09:03:19.952154 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0619 09:03:19.952157 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 09:03:19.952798 (+ 641us) server_negotiation.cc:407] Connection header received
0619 09:03:19.953288 (+ 490us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:03:19.953289 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:03:19.953301 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:03:19.953351 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:03:19.953914 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:03:19.954448 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:03:19.955460 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:03:19.955666 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:03:19.955722 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:03:19.957680 (+ 1958us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:03:19.957682 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:03:19.957682 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:03:19.957690 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:03:19.959151 (+ 1461us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:03:19.960818 (+ 1667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:03:19.960819 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:03:19.960820 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:03:19.960889 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:03:19.961105 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:03:19.961106 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:03:19.961107 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:03:19.961224 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:03:19.961279 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0619 09:03:19.961546 (+ 267us) server_negotiation.cc:299] Negotiation successful
0619 09:03:19.961625 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260619 09:03:19.962002 13125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:03:19.952096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44911 (local address 127.0.98.3:51531)
0619 09:03:19.952667 (+ 571us) negotiation.cc:107] Waiting for socket to connect
0619 09:03:19.952708 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 09:03:19.953188 (+ 480us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:03:19.953460 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:03:19.953463 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:03:19.953628 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0619 09:03:19.953820 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:03:19.953824 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:03:19.954528 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:03:19.954529 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:03:19.955380 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:03:19.955383 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:03:19.955750 (+ 367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:03:19.955751 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:03:19.955859 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:03:19.956375 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 09:03:19.956384 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0619 09:03:19.957590 (+ 1206us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 09:03:19.959283 (+ 1693us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:03:19.959289 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:03:19.959292 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:03:19.960739 (+ 1447us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:03:19.960945 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:03:19.960946 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:03:19.960947 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:03:19.961050 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:03:19.961278 (+ 228us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 09:03:19.961284 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 09:03:19.961428 (+ 144us) client_negotiation.cc:769] Sending connection context
0619 09:03:19.961515 (+ 87us) client_negotiation.cc:240] Negotiation successful
0619 09:03:19.961652 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":530,"thread_start_us":90,"threads_started":1}
I20260619 09:03:19.962524 13123 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44911
I20260619 09:03:19.962616 13123 heartbeater.cc:464] Registering TS with master...
I20260619 09:03:19.962792 13123 heartbeater.cc:511] Master 127.0.98.62:44911 requested a full tablet report, sending...
I20260619 09:03:19.963193 12639 ts_manager.cc:194] Registered new tserver with Master: 8bf3e72f7bc44976a8b60b1e5473c061 (127.0.98.3:39527)
I20260619 09:03:19.963546 12639 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:51531
I20260619 09:03:19.970484 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 09:03:19.971167 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ez5NB/build/release/bin/kudu with pid 12720
I20260619 09:03:19.976046 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ez5NB/build/release/bin/kudu with pid 12857
I20260619 09:03:19.979631 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ez5NB/build/release/bin/kudu with pid 12994
I20260619 09:03:19.984279 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ez5NB/build/release/bin/kudu with pid 12607
I20260619 09:03:19.988521 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ez5NB/build/release/bin/kudu
/tmp/dist-test-task4ez5NB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859658134510-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859658134510-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859658134510-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859658134510-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44911
--webserver_interface=127.0.98.62
--webserver_port=42341
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859658134510-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44911
--ranger_config_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859658134510-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:48805/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:48805/kms with env {KRB5CCNAME=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859658134510-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859658134510-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859658134510-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 09:03:20.066354 13126 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 09:03:20.066555 13126 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 09:03:20.066581 13126 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 09:03:20.067742 13126 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 09:03:20.067785 13126 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 09:03:20.067802 13126 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 09:03:20.067818 13126 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 09:03:20.067832 13126 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 09:03:20.069365 13126 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859658134510-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859658134510-392-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:48805/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44911
--ranger_config_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859658134510-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859658134510-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44911
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859658134510-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=42341
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13126
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859658134510-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 09:03:20.069613 13126 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 09:03:20.069880 13126 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 09:03:20.074484 13134 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:03:20.074540 13126 server_base.cc:1034] running on GCE node
W20260619 09:03:20.074513 13131 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:03:20.074558 13132 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:03:20.074947 13126 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 09:03:20.075222 13126 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 09:03:20.076382 13126 hybrid_clock.cc:648] HybridClock initialized: now 1781859800076369 us; error 27 us; skew 500 ppm
Jun 19 09:03:20 dist-test-slave-mtdr krb5kdc[12163](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859800, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 09:03:20.078783 13126 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 09:03:20.079246 13126 webserver.cc:460] Webserver started at http://127.0.98.62:42341/ using document root <none> and password file <none>
I20260619 09:03:20.079427 13126 fs_manager.cc:359] Metadata directory not provided
I20260619 09:03:20.079473 13126 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 19 09:03:20 dist-test-slave-mtdr krb5kdc[12163](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859800, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260619 09:03:20.156909 13126 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 09:03:20.157660 13142 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 09:03:20.157838 13126 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 09:03:20.157946 13126 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859658134510-392-0/minicluster-data/master-0/data,/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859658134510-392-0/minicluster-data/master-0/wal
uuid: "0b84830195484d8e96ffc8d8eb5ce0de"
format_stamp: "Formatted at 2026-06-19 09:03:18 on dist-test-slave-mtdr"
server_key: "a612ae6a3833955079c40bf4ecfff37d"
server_key_iv: "85028f6bfc716f8676554b52d9598d00"
server_key_version: "kuduclusterkey@0"
I20260619 09:03:20.158389 13126 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859658134510-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859658134510-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859658134510-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 09:03:20.166890 13126 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 09:03:20.168708 13126 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 09:03:20.168830 13126 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 09:03:20.172633 13126 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44911
I20260619 09:03:20.172665 13194 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44911 every 8 connection(s)
I20260619 09:03:20.173005 13126 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859658134510-392-0/minicluster-data/master-0/data/info.pb
I20260619 09:03:20.173992 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ez5NB/build/release/bin/kudu as pid 13126
I20260619 09:03:20.174216 13195 sys_catalog.cc:263] Verifying existing consensus state
I20260619 09:03:20.174268 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ez5NB/build/release/bin/kudu
/tmp/dist-test-task4ez5NB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859658134510-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859658134510-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859658134510-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859658134510-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:48805/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:40147
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:44911
--webserver_port=41875
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859658134510-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:48805/kms with env {KRB5CCNAME=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859658134510-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859658134510-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859658134510-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 09:03:20.174917 13195 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b84830195484d8e96ffc8d8eb5ce0de: Bootstrap starting.
I20260619 09:03:20.182379 13195 log.cc:824] T 00000000000000000000000000000000 P 0b84830195484d8e96ffc8d8eb5ce0de: Log is configured to *not* fsync() on all Append() calls
I20260619 09:03:20.184010 13195 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b84830195484d8e96ffc8d8eb5ce0de: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 09:03:20.184265 13195 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b84830195484d8e96ffc8d8eb5ce0de: Bootstrap complete.
I20260619 09:03:20.186367 13195 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0b84830195484d8e96ffc8d8eb5ce0de [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b84830195484d8e96ffc8d8eb5ce0de" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44911 } }
I20260619 09:03:20.186589 13195 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0b84830195484d8e96ffc8d8eb5ce0de [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b84830195484d8e96ffc8d8eb5ce0de, State: Initialized, Role: FOLLOWER
I20260619 09:03:20.186734 13195 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0b84830195484d8e96ffc8d8eb5ce0de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b84830195484d8e96ffc8d8eb5ce0de" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44911 } }
I20260619 09:03:20.186841 13195 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0b84830195484d8e96ffc8d8eb5ce0de [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 09:03:20.186884 13195 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0b84830195484d8e96ffc8d8eb5ce0de [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 09:03:20.186923 13195 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0b84830195484d8e96ffc8d8eb5ce0de [term 1 FOLLOWER]: Advancing to term 2
I20260619 09:03:20.187902 13195 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0b84830195484d8e96ffc8d8eb5ce0de [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b84830195484d8e96ffc8d8eb5ce0de" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44911 } }
I20260619 09:03:20.187995 13195 leader_election.cc:302] T 00000000000000000000000000000000 P 0b84830195484d8e96ffc8d8eb5ce0de [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0b84830195484d8e96ffc8d8eb5ce0de; no voters:
I20260619 09:03:20.188155 13195 leader_election.cc:288] T 00000000000000000000000000000000 P 0b84830195484d8e96ffc8d8eb5ce0de [CANDIDATE]: Term 2 election: Requested vote from peers
I20260619 09:03:20.188191 13199 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0b84830195484d8e96ffc8d8eb5ce0de [term 2 FOLLOWER]: Leader election won for term 2
I20260619 09:03:20.188287 13199 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0b84830195484d8e96ffc8d8eb5ce0de [term 2 LEADER]: Becoming Leader. State: Replica: 0b84830195484d8e96ffc8d8eb5ce0de, State: Running, Role: LEADER
I20260619 09:03:20.188403 13195 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0b84830195484d8e96ffc8d8eb5ce0de [sys.catalog]: configured and running, proceeding with master startup.
I20260619 09:03:20.188371 13199 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0b84830195484d8e96ffc8d8eb5ce0de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b84830195484d8e96ffc8d8eb5ce0de" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44911 } }
I20260619 09:03:20.188659 13200 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b84830195484d8e96ffc8d8eb5ce0de [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "0b84830195484d8e96ffc8d8eb5ce0de" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b84830195484d8e96ffc8d8eb5ce0de" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44911 } } }
I20260619 09:03:20.188748 13200 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b84830195484d8e96ffc8d8eb5ce0de [sys.catalog]: This master's current role is: LEADER
I20260619 09:03:20.188673 13201 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b84830195484d8e96ffc8d8eb5ce0de [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0b84830195484d8e96ffc8d8eb5ce0de. Latest consensus state: current_term: 2 leader_uuid: "0b84830195484d8e96ffc8d8eb5ce0de" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b84830195484d8e96ffc8d8eb5ce0de" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44911 } } }
I20260619 09:03:20.189000 13201 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b84830195484d8e96ffc8d8eb5ce0de [sys.catalog]: This master's current role is: LEADER
I20260619 09:03:20.189142 13203 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 09:03:20.189621 13203 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 09:03:20.189778 13203 catalog_manager.cc:1259] Loaded cluster ID: a54cc6f58a6841d2812e2d89720fa371
I20260619 09:03:20.189844 13203 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 09:03:20.189934 13195 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859658134510-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260619 09:03:20.193286 13203 catalog_manager.cc:1504] Loading token signing keys...
I20260619 09:03:20.193768 13203 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 0b84830195484d8e96ffc8d8eb5ce0de: Loaded TSK: 0
I20260619 09:03:20.194000 13203 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 09:03:20.275657 13197 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 09:03:20.275853 13197 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 09:03:20.275883 13197 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 09:03:20.277009 13197 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 09:03:20.277047 13197 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 09:03:20.277082 13197 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 09:03:20.278900 13197 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859658134510-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859658134510-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:48805/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859658134510-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:40147
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859658134510-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=41875
--tserver_master_addrs=127.0.98.62:44911
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13197
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859658134510-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 09:03:20.279124 13197 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 09:03:20.279331 13197 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 09:03:20.283914 13232 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:03:20.283926 13233 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:03:20.284240 13235 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:03:20.284469 13197 server_base.cc:1034] running on GCE node
I20260619 09:03:20.284647 13197 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 09:03:20.284862 13197 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 09:03:20.286068 13197 hybrid_clock.cc:648] HybridClock initialized: now 1781859800286054 us; error 25 us; skew 500 ppm
Jun 19 09:03:20 dist-test-slave-mtdr krb5kdc[12163](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859800, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 09:03:20.289175 13197 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 09:03:20.289700 13197 webserver.cc:460] Webserver started at http://127.0.98.1:41875/ using document root <none> and password file <none>
I20260619 09:03:20.289937 13197 fs_manager.cc:359] Metadata directory not provided
I20260619 09:03:20.290011 13197 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 19 09:03:20 dist-test-slave-mtdr krb5kdc[12163](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859800, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260619 09:03:20.315765 13197 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.000s
I20260619 09:03:20.316597 13243 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 09:03:20.316780 13197 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260619 09:03:20.316865 13197 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859658134510-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859658134510-392-0/minicluster-data/ts-0/wal
uuid: "412d20e190f64b59b233e4b7da26de6b"
format_stamp: "Formatted at 2026-06-19 09:03:19 on dist-test-slave-mtdr"
server_key: "96998d4b10e5983a4be1d57e99e8081e"
server_key_iv: "1bd7e0d82dac79e05ab93e4bd37ed610"
server_key_version: "kuduclusterkey@0"
I20260619 09:03:20.317286 13197 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859658134510-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859658134510-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859658134510-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 09:03:20.345450 13197 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 09:03:20.347232 13197 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 09:03:20.347374 13197 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 09:03:20.347563 13197 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 09:03:20.347895 13197 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 09:03:20.347939 13197 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 09:03:20.347967 13197 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 09:03:20.347991 13197 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 09:03:20.354007 13197 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40147
I20260619 09:03:20.354138 13356 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40147 every 8 connection(s)
I20260619 09:03:20.354411 13197 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859658134510-392-0/minicluster-data/ts-0/data/info.pb
Jun 19 09:03:20 dist-test-slave-mtdr krb5kdc[12163](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859800, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 09:03:20.363544 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ez5NB/build/release/bin/kudu as pid 13197
I20260619 09:03:20.363852 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ez5NB/build/release/bin/kudu
/tmp/dist-test-task4ez5NB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859658134510-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859658134510-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859658134510-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859658134510-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:48805/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:33713
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:44911
--webserver_port=42285
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859658134510-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:48805/kms with env {KRB5CCNAME=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859658134510-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859658134510-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859658134510-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 09:03:20.371594 13359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:03:20.355107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44911 (local address 127.0.98.1:45399)
0619 09:03:20.355261 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0619 09:03:20.355307 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 09:03:20.355921 (+ 614us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:03:20.356603 (+ 682us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:03:20.356607 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:03:20.356784 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0619 09:03:20.357024 (+ 240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:03:20.357032 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:03:20.358642 (+ 1610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:03:20.358643 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:03:20.359460 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:03:20.359462 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:03:20.360252 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:03:20.360253 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:03:20.360349 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:03:20.360801 (+ 452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 09:03:20.360816 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 09:03:20.362139 (+ 1323us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 09:03:20.369376 (+ 7237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:03:20.369384 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:03:20.369386 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:03:20.369698 (+ 312us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:03:20.369965 (+ 267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:03:20.369966 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:03:20.369967 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:03:20.370084 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:03:20.370898 (+ 814us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 09:03:20.370905 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 09:03:20.371084 (+ 179us) client_negotiation.cc:769] Sending connection context
0619 09:03:20.371181 (+ 97us) client_negotiation.cc:240] Negotiation successful
0619 09:03:20.371352 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":78,"threads_started":1}
I20260619 09:03:20.372355 13360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:03:20.355260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45399 (local address 127.0.98.62:44911)
0619 09:03:20.356132 (+ 872us) server_negotiation.cc:206] Beginning negotiation
0619 09:03:20.356141 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 09:03:20.356177 (+ 36us) server_negotiation.cc:407] Connection header received
0619 09:03:20.356324 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:03:20.356331 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:03:20.356386 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:03:20.356529 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:03:20.357152 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:03:20.358542 (+ 1390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:03:20.359542 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:03:20.360154 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:03:20.360260 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:03:20.362712 (+ 2452us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:03:20.362716 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:03:20.362716 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:03:20.362727 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:03:20.369252 (+ 6525us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:03:20.369790 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:03:20.369793 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:03:20.369793 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:03:20.369890 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:03:20.370380 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:03:20.370381 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:03:20.370382 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:03:20.370752 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:03:20.371931 (+ 1179us) server_negotiation.cc:1035] Waiting for connection context
0619 09:03:20.372012 (+ 81us) server_negotiation.cc:299] Negotiation successful
0619 09:03:20.372145 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":817,"thread_start_us":80,"threads_started":1}
I20260619 09:03:20.373020 13357 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44911
I20260619 09:03:20.373131 13357 heartbeater.cc:464] Registering TS with master...
I20260619 09:03:20.373335 13357 heartbeater.cc:511] Master 127.0.98.62:44911 requested a full tablet report, sending...
W20260619 09:03:20.374202 13357 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:44911 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260619 09:03:20.374279 13357 heartbeater.cc:464] Registering TS with master...
I20260619 09:03:20.374344 13357 heartbeater.cc:511] Master 127.0.98.62:44911 requested a full tablet report, sending...
I20260619 09:03:20.374673 13357 heartbeater.cc:464] Registering TS with master...
I20260619 09:03:20.374756 13357 heartbeater.cc:511] Master 127.0.98.62:44911 requested a full tablet report, sending...
W20260619 09:03:20.375082 13357 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 09:03:20.476691 13361 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 09:03:20.476907 13361 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 09:03:20.476940 13361 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 09:03:20.478113 13361 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 09:03:20.478159 13361 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 09:03:20.478195 13361 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 09:03:20.479737 13361 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859658134510-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859658134510-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:48805/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859658134510-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:33713
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859658134510-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=42285
--tserver_master_addrs=127.0.98.62:44911
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13361
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859658134510-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 09:03:20.479965 13361 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 09:03:20.480288 13361 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 09:03:20.485787 13367 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:03:20.485802 13369 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:03:20.485949 13366 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:03:20.486196 13361 server_base.cc:1034] running on GCE node
I20260619 09:03:20.486387 13361 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 09:03:20.486639 13361 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 09:03:20.487784 13361 hybrid_clock.cc:648] HybridClock initialized: now 1781859800487763 us; error 36 us; skew 500 ppm
Jun 19 09:03:20 dist-test-slave-mtdr krb5kdc[12163](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859800, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 09:03:20.490573 13361 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 09:03:20.491086 13361 webserver.cc:460] Webserver started at http://127.0.98.2:42285/ using document root <none> and password file <none>
I20260619 09:03:20.491287 13361 fs_manager.cc:359] Metadata directory not provided
I20260619 09:03:20.491348 13361 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 19 09:03:20 dist-test-slave-mtdr krb5kdc[12163](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859800, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260619 09:03:20.507918 13361 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 09:03:20.508701 13377 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 09:03:20.508875 13361 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 09:03:20.508980 13361 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859658134510-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859658134510-392-0/minicluster-data/ts-1/wal
uuid: "5712911e376c4b78b0d6c49a9495c911"
format_stamp: "Formatted at 2026-06-19 09:03:19 on dist-test-slave-mtdr"
server_key: "52f5d74bf3b84e8b42acf09563150d84"
server_key_iv: "7ebb1b22e43b108cf19129790b20c16a"
server_key_version: "kuduclusterkey@0"
I20260619 09:03:20.509400 13361 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859658134510-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859658134510-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859658134510-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 09:03:20.534116 13361 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 09:03:20.536131 13361 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 09:03:20.536267 13361 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 09:03:20.536515 13361 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 09:03:20.536785 13361 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 09:03:20.536814 13361 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 09:03:20.536834 13361 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 09:03:20.536846 13361 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 09:03:20.542727 13361 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33713
I20260619 09:03:20.543110 13361 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859658134510-392-0/minicluster-data/ts-1/data/info.pb
I20260619 09:03:20.543479 13490 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33713 every 8 connection(s)
Jun 19 09:03:20 dist-test-slave-mtdr krb5kdc[12163](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859800, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 09:03:20.549686 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ez5NB/build/release/bin/kudu as pid 13361
I20260619 09:03:20.549985 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ez5NB/build/release/bin/kudu
/tmp/dist-test-task4ez5NB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859658134510-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859658134510-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859658134510-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859658134510-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:48805/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:39527
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:44911
--webserver_port=43683
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859658134510-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:48805/kms with env {KRB5CCNAME=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859658134510-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859658134510-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859658134510-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 09:03:20.554807 13360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:03:20.543888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36557 (local address 127.0.98.62:44911)
0619 09:03:20.543938 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0619 09:03:20.543939 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 09:03:20.544192 (+ 253us) server_negotiation.cc:407] Connection header received
0619 09:03:20.544691 (+ 499us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:03:20.544692 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:03:20.544708 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:03:20.544766 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:03:20.545294 (+ 528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:03:20.545799 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:03:20.546773 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:03:20.547038 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:03:20.547095 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:03:20.551951 (+ 4856us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:03:20.551953 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:03:20.551953 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:03:20.551963 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:03:20.553522 (+ 1559us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:03:20.553928 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:03:20.553929 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:03:20.553930 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:03:20.553985 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:03:20.554194 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:03:20.554195 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:03:20.554195 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:03:20.554308 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:03:20.554362 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0619 09:03:20.554635 (+ 273us) server_negotiation.cc:299] Negotiation successful
0619 09:03:20.554716 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260619 09:03:20.555033 13493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:03:20.543924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44911 (local address 127.0.98.2:36557)
0619 09:03:20.544076 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0619 09:03:20.544118 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 09:03:20.544601 (+ 483us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:03:20.544837 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:03:20.544841 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:03:20.545016 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0619 09:03:20.545203 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:03:20.545207 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:03:20.545879 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:03:20.545880 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:03:20.546692 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:03:20.546695 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:03:20.547119 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:03:20.547120 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:03:20.547229 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:03:20.547682 (+ 453us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 09:03:20.547689 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0619 09:03:20.551825 (+ 4136us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 09:03:20.553600 (+ 1775us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:03:20.553608 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:03:20.553610 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:03:20.553823 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:03:20.554037 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:03:20.554038 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:03:20.554038 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:03:20.554137 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:03:20.554359 (+ 222us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 09:03:20.554366 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 09:03:20.554516 (+ 150us) client_negotiation.cc:769] Sending connection context
0619 09:03:20.554608 (+ 92us) client_negotiation.cc:240] Negotiation successful
0619 09:03:20.554745 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":72,"threads_started":1}
I20260619 09:03:20.555471 13491 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44911
I20260619 09:03:20.555558 13491 heartbeater.cc:464] Registering TS with master...
I20260619 09:03:20.555727 13491 heartbeater.cc:511] Master 127.0.98.62:44911 requested a full tablet report, sending...
W20260619 09:03:20.556325 13491 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:44911 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260619 09:03:20.556394 13491 heartbeater.cc:464] Registering TS with master...
I20260619 09:03:20.556461 13491 heartbeater.cc:511] Master 127.0.98.62:44911 requested a full tablet report, sending...
I20260619 09:03:20.556732 13491 heartbeater.cc:464] Registering TS with master...
I20260619 09:03:20.556802 13491 heartbeater.cc:511] Master 127.0.98.62:44911 requested a full tablet report, sending...
W20260619 09:03:20.557164 13491 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 09:03:20.666671 13494 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 09:03:20.666965 13494 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 09:03:20.667007 13494 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 09:03:20.668677 13494 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 09:03:20.668740 13494 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 09:03:20.668778 13494 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 09:03:20.670805 13494 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859658134510-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859658134510-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:48805/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859658134510-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:39527
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859658134510-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=43683
--tserver_master_addrs=127.0.98.62:44911
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13494
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859658134510-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 09:03:20.671080 13494 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 09:03:20.671346 13494 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 09:03:20.678653 13503 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:03:20.678699 13501 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:03:20.679066 13500 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:03:20.680173 13494 server_base.cc:1034] running on GCE node
I20260619 09:03:20.680390 13494 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 09:03:20.680646 13494 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 09:03:20.681881 13494 hybrid_clock.cc:648] HybridClock initialized: now 1781859800681858 us; error 38 us; skew 500 ppm
Jun 19 09:03:20 dist-test-slave-mtdr krb5kdc[12163](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859800, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 09:03:20.684427 13494 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 09:03:20.684845 13494 webserver.cc:460] Webserver started at http://127.0.98.3:43683/ using document root <none> and password file <none>
I20260619 09:03:20.685065 13494 fs_manager.cc:359] Metadata directory not provided
I20260619 09:03:20.685123 13494 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 19 09:03:20 dist-test-slave-mtdr krb5kdc[12163](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859800, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260619 09:03:20.698094 13494 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260619 09:03:20.698939 13511 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 09:03:20.699170 13494 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260619 09:03:20.699273 13494 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859658134510-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859658134510-392-0/minicluster-data/ts-2/wal
uuid: "8bf3e72f7bc44976a8b60b1e5473c061"
format_stamp: "Formatted at 2026-06-19 09:03:19 on dist-test-slave-mtdr"
server_key: "d126301ff7d8eeaa8f5ea2e0a9bab6f4"
server_key_iv: "bfbc19ae73becc1e23af4f4ab3fcb2a8"
server_key_version: "kuduclusterkey@0"
I20260619 09:03:20.699734 13494 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859658134510-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859658134510-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859658134510-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 09:03:20.721400 13494 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 09:03:20.723387 13494 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 09:03:20.723526 13494 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 09:03:20.723737 13494 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 09:03:20.724051 13494 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 09:03:20.724092 13494 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 09:03:20.724118 13494 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 09:03:20.724134 13494 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 09:03:20.730564 13494 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39527
I20260619 09:03:20.730631 13624 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39527 every 8 connection(s)
I20260619 09:03:20.731024 13494 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859658134510-392-0/minicluster-data/ts-2/data/info.pb
I20260619 09:03:20.736199 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ez5NB/build/release/bin/kudu as pid 13494
Jun 19 09:03:20 dist-test-slave-mtdr krb5kdc[12163](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859800, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 09:03:20.740360 13627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:03:20.731715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44911 (local address 127.0.98.3:60527)
0619 09:03:20.731876 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0619 09:03:20.731918 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 09:03:20.732407 (+ 489us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:03:20.732630 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:03:20.732634 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:03:20.732807 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0619 09:03:20.733002 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:03:20.733007 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:03:20.733660 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:03:20.733661 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:03:20.734485 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:03:20.734488 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:03:20.734870 (+ 382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:03:20.734871 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:03:20.734977 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:03:20.735473 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 09:03:20.735483 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 09:03:20.736994 (+ 1511us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 09:03:20.738721 (+ 1727us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:03:20.738728 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:03:20.738730 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:03:20.739072 (+ 342us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:03:20.739267 (+ 195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:03:20.739268 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:03:20.739268 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:03:20.739409 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:03:20.739699 (+ 290us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 09:03:20.739705 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 09:03:20.739884 (+ 179us) client_negotiation.cc:769] Sending connection context
0619 09:03:20.739984 (+ 100us) client_negotiation.cc:240] Negotiation successful
0619 09:03:20.740128 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":71,"threads_started":1}
I20260619 09:03:20.740795 13360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:03:20.731735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60527 (local address 127.0.98.62:44911)
0619 09:03:20.731784 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0619 09:03:20.731786 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:03:20.732007 (+ 221us) server_negotiation.cc:407] Connection header received
0619 09:03:20.732501 (+ 494us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:03:20.732502 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:03:20.732513 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:03:20.732563 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:03:20.733091 (+ 528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:03:20.733581 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:03:20.734562 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:03:20.734766 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:03:20.734842 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:03:20.737105 (+ 2263us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:03:20.737107 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:03:20.737107 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:03:20.737115 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:03:20.738633 (+ 1518us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:03:20.739135 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:03:20.739137 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:03:20.739137 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:03:20.739198 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:03:20.739482 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:03:20.739483 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:03:20.739484 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:03:20.739619 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:03:20.739674 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0619 09:03:20.740629 (+ 955us) server_negotiation.cc:299] Negotiation successful
0619 09:03:20.740705 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260619 09:03:20.741530 13625 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44911
I20260619 09:03:20.741638 13625 heartbeater.cc:464] Registering TS with master...
I20260619 09:03:20.741822 13625 heartbeater.cc:511] Master 127.0.98.62:44911 requested a full tablet report, sending...
W20260619 09:03:20.742384 13625 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:44911 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260619 09:03:20.742455 13625 heartbeater.cc:464] Registering TS with master...
I20260619 09:03:20.742517 13625 heartbeater.cc:511] Master 127.0.98.62:44911 requested a full tablet report, sending...
I20260619 09:03:20.742786 13625 heartbeater.cc:464] Registering TS with master...
I20260619 09:03:20.742893 13625 heartbeater.cc:511] Master 127.0.98.62:44911 requested a full tablet report, sending...
W20260619 09:03:20.743165 13625 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260619 09:03:20.746547 13628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:03:20.736779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56862 (local address 127.0.98.62:44911)
0619 09:03:20.737382 (+ 603us) server_negotiation.cc:206] Beginning negotiation
0619 09:03:20.737384 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:03:20.737530 (+ 146us) server_negotiation.cc:407] Connection header received
0619 09:03:20.737974 (+ 444us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:03:20.737975 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:03:20.737987 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:03:20.739160 (+ 1173us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:03:20.739920 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:03:20.741056 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:03:20.742204 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:03:20.742597 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:03:20.742637 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:03:20.743946 (+ 1309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:03:20.743947 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:03:20.743948 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:03:20.743955 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:03:20.745438 (+ 1483us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:03:20.745754 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:03:20.745755 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:03:20.745755 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:03:20.745815 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:03:20.746032 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:03:20.746033 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:03:20.746034 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:03:20.746164 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:03:20.746328 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0619 09:03:20.746365 (+ 37us) server_negotiation.cc:299] Negotiation successful
0619 09:03:20.746456 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":583,"thread_start_us":60,"threads_started":1}
Jun 19 09:03:21 dist-test-slave-mtdr krb5kdc[12163](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781859801, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 09:03:21.375269 13357 heartbeater.cc:464] Registering TS with master...
I20260619 09:03:21.375442 13357 heartbeater.cc:511] Master 127.0.98.62:44911 requested a full tablet report, sending...
I20260619 09:03:21.375813 13159 ts_manager.cc:194] Registered new tserver with Master: 412d20e190f64b59b233e4b7da26de6b (127.0.98.1:40147)
I20260619 09:03:21.376667 13159 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:45399
I20260619 09:03:21.557343 13491 heartbeater.cc:464] Registering TS with master...
I20260619 09:03:21.557498 13491 heartbeater.cc:511] Master 127.0.98.62:44911 requested a full tablet report, sending...
I20260619 09:03:21.558084 13159 ts_manager.cc:194] Registered new tserver with Master: 5712911e376c4b78b0d6c49a9495c911 (127.0.98.2:33713)
I20260619 09:03:21.558682 13159 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:36557
Jun 19 09:03:21 dist-test-slave-mtdr krb5kdc[12163](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859801, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260619 09:03:21.743340 13625 heartbeater.cc:464] Registering TS with master...
I20260619 09:03:21.743490 13625 heartbeater.cc:511] Master 127.0.98.62:44911 requested a full tablet report, sending...
I20260619 09:03:21.743785 13159 ts_manager.cc:194] Registered new tserver with Master: 8bf3e72f7bc44976a8b60b1e5473c061 (127.0.98.3:39527)
I20260619 09:03:21.744278 13159 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:60527
I20260619 09:03:21.748829 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 09:03:21.757157 13647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:03:21.749798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59488 (local address 127.0.98.62:44911)
0619 09:03:21.749927 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0619 09:03:21.749930 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 09:03:21.750060 (+ 130us) server_negotiation.cc:407] Connection header received
0619 09:03:21.750124 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:03:21.750125 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:03:21.750137 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:03:21.750203 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:03:21.750884 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:03:21.751615 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:03:21.752623 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:03:21.752907 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:03:21.752965 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:03:21.754362 (+ 1397us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:03:21.754364 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:03:21.754364 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:03:21.754371 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:03:21.756055 (+ 1684us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:03:21.756427 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:03:21.756428 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:03:21.756429 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:03:21.756486 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:03:21.756668 (+ 182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:03:21.756668 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:03:21.756668 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:03:21.756781 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:03:21.756904 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0619 09:03:21.756992 (+ 88us) server_negotiation.cc:299] Negotiation successful
0619 09:03:21.757067 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":71,"threads_started":1}
I20260619 09:03:21.759469 13159 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59488:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 09:03:21.761649 13159 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 09:03:21.769289 13654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:03:21.764492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39527 (local address 127.0.0.1:52508)
0619 09:03:21.764826 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0619 09:03:21.764864 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 09:03:21.764936 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:03:21.765374 (+ 438us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:03:21.765377 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:03:21.765380 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 09:03:21.765444 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:03:21.765446 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:03:21.766770 (+ 1324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:03:21.766771 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:03:21.768058 (+ 1287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:03:21.768062 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:03:21.769009 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:03:21.769010 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:03:21.769089 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:03:21.769105 (+ 16us) client_negotiation.cc:769] Sending connection context
0619 09:03:21.769155 (+ 50us) client_negotiation.cc:240] Negotiation successful
0619 09:03:21.769205 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":78,"threads_started":1}
I20260619 09:03:21.769938 13657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:03:21.765065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52508 (local address 127.0.98.3:39527)
0619 09:03:21.765196 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0619 09:03:21.765200 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 09:03:21.765211 (+ 11us) server_negotiation.cc:407] Connection header received
0619 09:03:21.765245 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:03:21.765247 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:03:21.765262 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:03:21.765520 (+ 258us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 09:03:21.765588 (+ 68us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:03:21.766670 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:03:21.768159 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:03:21.768890 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:03:21.769696 (+ 806us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:03:21.769785 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0619 09:03:21.769810 (+ 25us) server_negotiation.cc:299] Negotiation successful
0619 09:03:21.769847 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96,"thread_start_us":52,"threads_started":1}
I20260619 09:03:21.770499 13653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:03:21.764502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40147 (local address 127.0.0.1:49754)
0619 09:03:21.765541 (+ 1039us) negotiation.cc:107] Waiting for socket to connect
0619 09:03:21.765550 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 09:03:21.765595 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:03:21.765825 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:03:21.765825 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:03:21.765827 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 09:03:21.765864 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:03:21.765865 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:03:21.767313 (+ 1448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:03:21.767314 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:03:21.768438 (+ 1124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:03:21.768441 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:03:21.770102 (+ 1661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:03:21.770103 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:03:21.770335 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:03:21.770339 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 09:03:21.770384 (+ 45us) client_negotiation.cc:240] Negotiation successful
0619 09:03:21.770431 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1023,"thread_start_us":85,"threads_started":1}
I20260619 09:03:21.770764 13656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:03:21.764906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49754 (local address 127.0.98.1:40147)
0619 09:03:21.765489 (+ 583us) server_negotiation.cc:206] Beginning negotiation
0619 09:03:21.765493 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 09:03:21.765640 (+ 147us) server_negotiation.cc:407] Connection header received
0619 09:03:21.765673 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:03:21.765675 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:03:21.765689 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:03:21.765757 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 09:03:21.765925 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:03:21.766973 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:03:21.768560 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:03:21.769946 (+ 1386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:03:21.770004 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:03:21.770072 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0619 09:03:21.770657 (+ 585us) server_negotiation.cc:299] Negotiation successful
0619 09:03:21.770697 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":521,"thread_start_us":75,"threads_started":1}
I20260619 09:03:21.770766 13548 tablet_service.cc:1467] Processing CreateTablet for tablet 1b668d06ca044da2ba55145eb2ca6115 (DEFAULT_TABLE table=test-table [id=6c529532bb374f40ac50149f9fba1b50]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 09:03:21.771173 13548 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b668d06ca044da2ba55145eb2ca6115. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 09:03:21.771368 13290 tablet_service.cc:1467] Processing CreateTablet for tablet 1b668d06ca044da2ba55145eb2ca6115 (DEFAULT_TABLE table=test-table [id=6c529532bb374f40ac50149f9fba1b50]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 09:03:21.771689 13290 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b668d06ca044da2ba55145eb2ca6115. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 09:03:21.775660 13658 tablet_bootstrap.cc:492] T 1b668d06ca044da2ba55145eb2ca6115 P 412d20e190f64b59b233e4b7da26de6b: Bootstrap starting.
I20260619 09:03:21.776266 13658 tablet_bootstrap.cc:654] T 1b668d06ca044da2ba55145eb2ca6115 P 412d20e190f64b59b233e4b7da26de6b: Neither blocks nor log segments found. Creating new log.
I20260619 09:03:21.776558 13658 log.cc:824] T 1b668d06ca044da2ba55145eb2ca6115 P 412d20e190f64b59b233e4b7da26de6b: Log is configured to *not* fsync() on all Append() calls
I20260619 09:03:21.777343 13658 tablet_bootstrap.cc:492] T 1b668d06ca044da2ba55145eb2ca6115 P 412d20e190f64b59b233e4b7da26de6b: No bootstrap required, opened a new log
I20260619 09:03:21.777418 13658 ts_tablet_manager.cc:1397] T 1b668d06ca044da2ba55145eb2ca6115 P 412d20e190f64b59b233e4b7da26de6b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 09:03:21.780037 13658 raft_consensus.cc:348] T 1b668d06ca044da2ba55145eb2ca6115 P 412d20e190f64b59b233e4b7da26de6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bf3e72f7bc44976a8b60b1e5473c061" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39527 } } peers { permanent_uuid: "5712911e376c4b78b0d6c49a9495c911" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33713 } } peers { permanent_uuid: "412d20e190f64b59b233e4b7da26de6b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40147 } }
I20260619 09:03:21.780194 13658 raft_consensus.cc:374] T 1b668d06ca044da2ba55145eb2ca6115 P 412d20e190f64b59b233e4b7da26de6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 09:03:21.780223 13658 raft_consensus.cc:729] T 1b668d06ca044da2ba55145eb2ca6115 P 412d20e190f64b59b233e4b7da26de6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 412d20e190f64b59b233e4b7da26de6b, State: Initialized, Role: FOLLOWER
I20260619 09:03:21.780344 13658 consensus_queue.cc:260] T 1b668d06ca044da2ba55145eb2ca6115 P 412d20e190f64b59b233e4b7da26de6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bf3e72f7bc44976a8b60b1e5473c061" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39527 } } peers { permanent_uuid: "5712911e376c4b78b0d6c49a9495c911" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33713 } } peers { permanent_uuid: "412d20e190f64b59b233e4b7da26de6b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40147 } }
I20260619 09:03:21.780596 13658 ts_tablet_manager.cc:1428] T 1b668d06ca044da2ba55145eb2ca6115 P 412d20e190f64b59b233e4b7da26de6b: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260619 09:03:21.780711 13357 heartbeater.cc:503] Master 127.0.98.62:44911 was elected leader, sending a full tablet report...
I20260619 09:03:21.782945 13659 tablet_bootstrap.cc:492] T 1b668d06ca044da2ba55145eb2ca6115 P 8bf3e72f7bc44976a8b60b1e5473c061: Bootstrap starting.
I20260619 09:03:21.782977 13654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:03:21.764702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33713 (local address 127.0.0.1:52928)
0619 09:03:21.769569 (+ 4867us) negotiation.cc:107] Waiting for socket to connect
0619 09:03:21.769580 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 09:03:21.769622 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:03:21.779070 (+ 9448us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:03:21.779071 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:03:21.779073 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 09:03:21.779116 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:03:21.779118 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:03:21.780468 (+ 1350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:03:21.780470 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:03:21.781615 (+ 1145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:03:21.781618 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:03:21.782708 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:03:21.782710 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:03:21.782775 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:03:21.782778 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 09:03:21.782843 (+ 65us) client_negotiation.cc:240] Negotiation successful
0619 09:03:21.782900 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4857,"thread_start_us":71,"threads_started":1}
I20260619 09:03:21.783268 13655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:03:21.764803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52928 (local address 127.0.98.2:33713)
0619 09:03:21.778916 (+ 14113us) server_negotiation.cc:206] Beginning negotiation
0619 09:03:21.778920 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 09:03:21.778933 (+ 13us) server_negotiation.cc:407] Connection header received
0619 09:03:21.778968 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:03:21.778969 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:03:21.778985 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:03:21.779071 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 09:03:21.779203 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:03:21.780242 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:03:21.781806 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:03:21.782539 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:03:21.782608 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:03:21.782676 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0619 09:03:21.783146 (+ 470us) server_negotiation.cc:299] Negotiation successful
0619 09:03:21.783189 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14062,"thread_start_us":38,"threads_started":1}
I20260619 09:03:21.783591 13659 tablet_bootstrap.cc:654] T 1b668d06ca044da2ba55145eb2ca6115 P 8bf3e72f7bc44976a8b60b1e5473c061: Neither blocks nor log segments found. Creating new log.
I20260619 09:03:21.783874 13659 log.cc:824] T 1b668d06ca044da2ba55145eb2ca6115 P 8bf3e72f7bc44976a8b60b1e5473c061: Log is configured to *not* fsync() on all Append() calls
I20260619 09:03:21.783963 13420 tablet_service.cc:1467] Processing CreateTablet for tablet 1b668d06ca044da2ba55145eb2ca6115 (DEFAULT_TABLE table=test-table [id=6c529532bb374f40ac50149f9fba1b50]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 09:03:21.784289 13420 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b668d06ca044da2ba55145eb2ca6115. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 09:03:21.787210 13659 tablet_bootstrap.cc:492] T 1b668d06ca044da2ba55145eb2ca6115 P 8bf3e72f7bc44976a8b60b1e5473c061: No bootstrap required, opened a new log
I20260619 09:03:21.787322 13659 ts_tablet_manager.cc:1397] T 1b668d06ca044da2ba55145eb2ca6115 P 8bf3e72f7bc44976a8b60b1e5473c061: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260619 09:03:21.787937 13663 tablet_bootstrap.cc:492] T 1b668d06ca044da2ba55145eb2ca6115 P 5712911e376c4b78b0d6c49a9495c911: Bootstrap starting.
I20260619 09:03:21.788635 13663 tablet_bootstrap.cc:654] T 1b668d06ca044da2ba55145eb2ca6115 P 5712911e376c4b78b0d6c49a9495c911: Neither blocks nor log segments found. Creating new log.
I20260619 09:03:21.788928 13663 log.cc:824] T 1b668d06ca044da2ba55145eb2ca6115 P 5712911e376c4b78b0d6c49a9495c911: Log is configured to *not* fsync() on all Append() calls
I20260619 09:03:21.789764 13663 tablet_bootstrap.cc:492] T 1b668d06ca044da2ba55145eb2ca6115 P 5712911e376c4b78b0d6c49a9495c911: No bootstrap required, opened a new log
I20260619 09:03:21.789860 13663 ts_tablet_manager.cc:1397] T 1b668d06ca044da2ba55145eb2ca6115 P 5712911e376c4b78b0d6c49a9495c911: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260619 09:03:21.790316 13659 raft_consensus.cc:348] T 1b668d06ca044da2ba55145eb2ca6115 P 8bf3e72f7bc44976a8b60b1e5473c061 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bf3e72f7bc44976a8b60b1e5473c061" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39527 } } peers { permanent_uuid: "5712911e376c4b78b0d6c49a9495c911" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33713 } } peers { permanent_uuid: "412d20e190f64b59b233e4b7da26de6b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40147 } }
I20260619 09:03:21.790531 13659 raft_consensus.cc:374] T 1b668d06ca044da2ba55145eb2ca6115 P 8bf3e72f7bc44976a8b60b1e5473c061 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 09:03:21.790601 13659 raft_consensus.cc:729] T 1b668d06ca044da2ba55145eb2ca6115 P 8bf3e72f7bc44976a8b60b1e5473c061 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8bf3e72f7bc44976a8b60b1e5473c061, State: Initialized, Role: FOLLOWER
I20260619 09:03:21.790745 13659 consensus_queue.cc:260] T 1b668d06ca044da2ba55145eb2ca6115 P 8bf3e72f7bc44976a8b60b1e5473c061 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bf3e72f7bc44976a8b60b1e5473c061" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39527 } } peers { permanent_uuid: "5712911e376c4b78b0d6c49a9495c911" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33713 } } peers { permanent_uuid: "412d20e190f64b59b233e4b7da26de6b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40147 } }
I20260619 09:03:21.791069 13659 ts_tablet_manager.cc:1428] T 1b668d06ca044da2ba55145eb2ca6115 P 8bf3e72f7bc44976a8b60b1e5473c061: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260619 09:03:21.791208 13625 heartbeater.cc:503] Master 127.0.98.62:44911 was elected leader, sending a full tablet report...
I20260619 09:03:21.792627 13663 raft_consensus.cc:348] T 1b668d06ca044da2ba55145eb2ca6115 P 5712911e376c4b78b0d6c49a9495c911 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bf3e72f7bc44976a8b60b1e5473c061" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39527 } } peers { permanent_uuid: "5712911e376c4b78b0d6c49a9495c911" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33713 } } peers { permanent_uuid: "412d20e190f64b59b233e4b7da26de6b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40147 } }
I20260619 09:03:21.792779 13663 raft_consensus.cc:374] T 1b668d06ca044da2ba55145eb2ca6115 P 5712911e376c4b78b0d6c49a9495c911 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 09:03:21.792810 13663 raft_consensus.cc:729] T 1b668d06ca044da2ba55145eb2ca6115 P 5712911e376c4b78b0d6c49a9495c911 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5712911e376c4b78b0d6c49a9495c911, State: Initialized, Role: FOLLOWER
I20260619 09:03:21.792997 13663 consensus_queue.cc:260] T 1b668d06ca044da2ba55145eb2ca6115 P 5712911e376c4b78b0d6c49a9495c911 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bf3e72f7bc44976a8b60b1e5473c061" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39527 } } peers { permanent_uuid: "5712911e376c4b78b0d6c49a9495c911" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33713 } } peers { permanent_uuid: "412d20e190f64b59b233e4b7da26de6b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40147 } }
I20260619 09:03:21.793252 13663 ts_tablet_manager.cc:1428] T 1b668d06ca044da2ba55145eb2ca6115 P 5712911e376c4b78b0d6c49a9495c911: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260619 09:03:21.793290 13491 heartbeater.cc:503] Master 127.0.98.62:44911 was elected leader, sending a full tablet report...
W20260619 09:03:21.794411 13492 tablet.cc:2369] T 1b668d06ca044da2ba55145eb2ca6115 P 5712911e376c4b78b0d6c49a9495c911: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 19 09:03:21 dist-test-slave-mtdr krb5kdc[12163](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859801, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
W20260619 09:03:21.855840 13358 tablet.cc:2369] T 1b668d06ca044da2ba55145eb2ca6115 P 412d20e190f64b59b233e4b7da26de6b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 09:03:21.983244 13626 tablet.cc:2369] T 1b668d06ca044da2ba55145eb2ca6115 P 8bf3e72f7bc44976a8b60b1e5473c061: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 09:03:22.040966 13666 raft_consensus.cc:482] T 1b668d06ca044da2ba55145eb2ca6115 P 5712911e376c4b78b0d6c49a9495c911 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 09:03:22.041119 13666 raft_consensus.cc:504] T 1b668d06ca044da2ba55145eb2ca6115 P 5712911e376c4b78b0d6c49a9495c911 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bf3e72f7bc44976a8b60b1e5473c061" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39527 } } peers { permanent_uuid: "5712911e376c4b78b0d6c49a9495c911" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33713 } } peers { permanent_uuid: "412d20e190f64b59b233e4b7da26de6b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40147 } }
I20260619 09:03:22.041533 13666 leader_election.cc:288] T 1b668d06ca044da2ba55145eb2ca6115 P 5712911e376c4b78b0d6c49a9495c911 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8bf3e72f7bc44976a8b60b1e5473c061 (127.0.98.3:39527), 412d20e190f64b59b233e4b7da26de6b (127.0.98.1:40147)
I20260619 09:03:22.045812 13656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:03:22.041919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46173 (local address 127.0.98.1:40147)
0619 09:03:22.041960 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0619 09:03:22.041962 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:03:22.042108 (+ 146us) server_negotiation.cc:407] Connection header received
0619 09:03:22.042214 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:03:22.042214 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:03:22.042226 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:03:22.042255 (+ 29us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 09:03:22.042526 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:03:22.043310 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:03:22.044820 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:03:22.045412 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:03:22.045457 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:03:22.045468 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0619 09:03:22.045697 (+ 229us) server_negotiation.cc:299] Negotiation successful
0619 09:03:22.045740 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260619 09:03:22.045812 13657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:03:22.041797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42855 (local address 127.0.98.3:39527)
0619 09:03:22.041842 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0619 09:03:22.041844 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:03:22.042111 (+ 267us) server_negotiation.cc:407] Connection header received
0619 09:03:22.042214 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:03:22.042215 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:03:22.042226 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:03:22.042255 (+ 29us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 09:03:22.042525 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:03:22.043311 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:03:22.044821 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:03:22.045412 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:03:22.045457 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:03:22.045468 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0619 09:03:22.045697 (+ 229us) server_negotiation.cc:299] Negotiation successful
0619 09:03:22.045740 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260619 09:03:22.046061 13669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:03:22.041715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40147 (local address 127.0.98.2:46173)
0619 09:03:22.042025 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0619 09:03:22.042036 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 09:03:22.042138 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:03:22.042321 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:03:22.042321 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:03:22.042324 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 09:03:22.042449 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:03:22.042450 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:03:22.043409 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:03:22.043410 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:03:22.044715 (+ 1305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:03:22.044719 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:03:22.045547 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:03:22.045548 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:03:22.045614 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:03:22.045618 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 09:03:22.045922 (+ 304us) client_negotiation.cc:240] Negotiation successful
0619 09:03:22.045973 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"mutex_wait_us":112,"thread_start_us":89,"threads_started":1}
I20260619 09:03:22.046379 13670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:03:22.041715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39527 (local address 127.0.98.2:42855)
0619 09:03:22.042059 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0619 09:03:22.042069 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 09:03:22.042138 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:03:22.042321 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:03:22.042321 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:03:22.042323 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 09:03:22.042448 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:03:22.042450 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:03:22.043409 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:03:22.043410 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:03:22.044715 (+ 1305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:03:22.044719 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:03:22.045529 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:03:22.045530 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:03:22.045614 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:03:22.045618 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 09:03:22.046254 (+ 636us) client_negotiation.cc:240] Negotiation successful
0619 09:03:22.046291 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":333,"mutex_wait_us":248,"thread_start_us":78,"threads_started":1}
I20260619 09:03:22.046342 13311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b668d06ca044da2ba55145eb2ca6115" candidate_uuid: "5712911e376c4b78b0d6c49a9495c911" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "412d20e190f64b59b233e4b7da26de6b" is_pre_election: true
I20260619 09:03:22.046509 13311 raft_consensus.cc:2454] T 1b668d06ca044da2ba55145eb2ca6115 P 412d20e190f64b59b233e4b7da26de6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5712911e376c4b78b0d6c49a9495c911 in term 0.
I20260619 09:03:22.046658 13579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b668d06ca044da2ba55145eb2ca6115" candidate_uuid: "5712911e376c4b78b0d6c49a9495c911" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8bf3e72f7bc44976a8b60b1e5473c061" is_pre_election: true
I20260619 09:03:22.046765 13379 leader_election.cc:302] T 1b668d06ca044da2ba55145eb2ca6115 P 5712911e376c4b78b0d6c49a9495c911 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 412d20e190f64b59b233e4b7da26de6b, 5712911e376c4b78b0d6c49a9495c911; no voters:
I20260619 09:03:22.046799 13579 raft_consensus.cc:2454] T 1b668d06ca044da2ba55145eb2ca6115 P 8bf3e72f7bc44976a8b60b1e5473c061 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5712911e376c4b78b0d6c49a9495c911 in term 0.
I20260619 09:03:22.046902 13666 raft_consensus.cc:2781] T 1b668d06ca044da2ba55145eb2ca6115 P 5712911e376c4b78b0d6c49a9495c911 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 09:03:22.046945 13666 raft_consensus.cc:482] T 1b668d06ca044da2ba55145eb2ca6115 P 5712911e376c4b78b0d6c49a9495c911 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 09:03:22.046969 13666 raft_consensus.cc:3037] T 1b668d06ca044da2ba55145eb2ca6115 P 5712911e376c4b78b0d6c49a9495c911 [term 0 FOLLOWER]: Advancing to term 1
I20260619 09:03:22.047773 13666 raft_consensus.cc:504] T 1b668d06ca044da2ba55145eb2ca6115 P 5712911e376c4b78b0d6c49a9495c911 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bf3e72f7bc44976a8b60b1e5473c061" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39527 } } peers { permanent_uuid: "5712911e376c4b78b0d6c49a9495c911" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33713 } } peers { permanent_uuid: "412d20e190f64b59b233e4b7da26de6b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40147 } }
I20260619 09:03:22.047894 13666 leader_election.cc:288] T 1b668d06ca044da2ba55145eb2ca6115 P 5712911e376c4b78b0d6c49a9495c911 [CANDIDATE]: Term 1 election: Requested vote from peers 8bf3e72f7bc44976a8b60b1e5473c061 (127.0.98.3:39527), 412d20e190f64b59b233e4b7da26de6b (127.0.98.1:40147)
I20260619 09:03:22.048077 13579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b668d06ca044da2ba55145eb2ca6115" candidate_uuid: "5712911e376c4b78b0d6c49a9495c911" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8bf3e72f7bc44976a8b60b1e5473c061"
I20260619 09:03:22.048134 13311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b668d06ca044da2ba55145eb2ca6115" candidate_uuid: "5712911e376c4b78b0d6c49a9495c911" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "412d20e190f64b59b233e4b7da26de6b"
I20260619 09:03:22.048149 13579 raft_consensus.cc:3037] T 1b668d06ca044da2ba55145eb2ca6115 P 8bf3e72f7bc44976a8b60b1e5473c061 [term 0 FOLLOWER]: Advancing to term 1
I20260619 09:03:22.048195 13311 raft_consensus.cc:3037] T 1b668d06ca044da2ba55145eb2ca6115 P 412d20e190f64b59b233e4b7da26de6b [term 0 FOLLOWER]: Advancing to term 1
I20260619 09:03:22.048976 13579 raft_consensus.cc:2454] T 1b668d06ca044da2ba55145eb2ca6115 P 8bf3e72f7bc44976a8b60b1e5473c061 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5712911e376c4b78b0d6c49a9495c911 in term 1.
I20260619 09:03:22.048976 13311 raft_consensus.cc:2454] T 1b668d06ca044da2ba55145eb2ca6115 P 412d20e190f64b59b233e4b7da26de6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5712911e376c4b78b0d6c49a9495c911 in term 1.
I20260619 09:03:22.049150 13379 leader_election.cc:302] T 1b668d06ca044da2ba55145eb2ca6115 P 5712911e376c4b78b0d6c49a9495c911 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 412d20e190f64b59b233e4b7da26de6b, 5712911e376c4b78b0d6c49a9495c911; no voters:
I20260619 09:03:22.049306 13666 raft_consensus.cc:2781] T 1b668d06ca044da2ba55145eb2ca6115 P 5712911e376c4b78b0d6c49a9495c911 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 09:03:22.049458 13666 raft_consensus.cc:686] T 1b668d06ca044da2ba55145eb2ca6115 P 5712911e376c4b78b0d6c49a9495c911 [term 1 LEADER]: Becoming Leader. State: Replica: 5712911e376c4b78b0d6c49a9495c911, State: Running, Role: LEADER
I20260619 09:03:22.049583 13666 consensus_queue.cc:237] T 1b668d06ca044da2ba55145eb2ca6115 P 5712911e376c4b78b0d6c49a9495c911 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bf3e72f7bc44976a8b60b1e5473c061" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39527 } } peers { permanent_uuid: "5712911e376c4b78b0d6c49a9495c911" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33713 } } peers { permanent_uuid: "412d20e190f64b59b233e4b7da26de6b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40147 } }
I20260619 09:03:22.050657 13159 catalog_manager.cc:5495] T 1b668d06ca044da2ba55145eb2ca6115 P 5712911e376c4b78b0d6c49a9495c911 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5712911e376c4b78b0d6c49a9495c911 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "5712911e376c4b78b0d6c49a9495c911" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bf3e72f7bc44976a8b60b1e5473c061" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39527 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5712911e376c4b78b0d6c49a9495c911" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33713 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "412d20e190f64b59b233e4b7da26de6b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40147 } health_report { overall_health: UNKNOWN } } }
I20260619 09:03:22.124991 13655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:03:22.122239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52944 (local address 127.0.98.2:33713)
0619 09:03:22.122281 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0619 09:03:22.122283 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:03:22.122292 (+ 9us) server_negotiation.cc:407] Connection header received
0619 09:03:22.122355 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:03:22.122355 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:03:22.122374 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:03:22.122432 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 09:03:22.122626 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:03:22.123168 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:03:22.124250 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:03:22.124432 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:03:22.124478 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:03:22.124671 (+ 193us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 09:03:22.124773 (+ 102us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 09:03:22.124847 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0619 09:03:22.124892 (+ 45us) server_negotiation.cc:299] Negotiation successful
0619 09:03:22.124910 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260619 09:03:22.126942 13311 raft_consensus.cc:1261] T 1b668d06ca044da2ba55145eb2ca6115 P 412d20e190f64b59b233e4b7da26de6b [term 1 FOLLOWER]: Refusing update from remote peer 5712911e376c4b78b0d6c49a9495c911: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 09:03:22.126976 13579 raft_consensus.cc:1261] T 1b668d06ca044da2ba55145eb2ca6115 P 8bf3e72f7bc44976a8b60b1e5473c061 [term 1 FOLLOWER]: Refusing update from remote peer 5712911e376c4b78b0d6c49a9495c911: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 09:03:22.127327 13671 consensus_queue.cc:1035] T 1b668d06ca044da2ba55145eb2ca6115 P 5712911e376c4b78b0d6c49a9495c911 [LEADER]: Connected to new peer: Peer: permanent_uuid: "412d20e190f64b59b233e4b7da26de6b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 09:03:22.127451 13666 consensus_queue.cc:1035] T 1b668d06ca044da2ba55145eb2ca6115 P 5712911e376c4b78b0d6c49a9495c911 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8bf3e72f7bc44976a8b60b1e5473c061" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 09:03:22.130520 13681 mvcc.cc:205] Tried to move back new op lower bound from 7298497749509095424 to 7298497749195771904. Current Snapshot: MvccSnapshot[applied={T|T < 7298497749509095424}]
I20260619 09:03:22.132984 13159 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59488:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 09:03:22.133075 13159 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59488:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 09:03:22.133821 13159 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0b84830195484d8e96ffc8d8eb5ce0de: Sending DeleteTablet for 3 replicas of tablet 1b668d06ca044da2ba55145eb2ca6115
I20260619 09:03:22.134137 13290 tablet_service.cc:1514] Processing DeleteTablet for tablet 1b668d06ca044da2ba55145eb2ca6115 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 09:03:22 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49754
I20260619 09:03:22.134572 13548 tablet_service.cc:1514] Processing DeleteTablet for tablet 1b668d06ca044da2ba55145eb2ca6115 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 09:03:22 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52508
I20260619 09:03:22.135075 13420 tablet_service.cc:1514] Processing DeleteTablet for tablet 1b668d06ca044da2ba55145eb2ca6115 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 09:03:22 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52928
I20260619 09:03:22.135265 13690 tablet_replica.cc:330] T 1b668d06ca044da2ba55145eb2ca6115 P 412d20e190f64b59b233e4b7da26de6b: stopping tablet replica
I20260619 09:03:22.135303 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ez5NB/build/release/bin/kudu with pid 13197
I20260619 09:03:22.135355 13690 raft_consensus.cc:2229] T 1b668d06ca044da2ba55145eb2ca6115 P 412d20e190f64b59b233e4b7da26de6b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 09:03:22.136101 13692 tablet_replica.cc:330] T 1b668d06ca044da2ba55145eb2ca6115 P 5712911e376c4b78b0d6c49a9495c911: stopping tablet replica
I20260619 09:03:22.136181 13692 raft_consensus.cc:2229] T 1b668d06ca044da2ba55145eb2ca6115 P 5712911e376c4b78b0d6c49a9495c911 [term 1 LEADER]: Raft consensus shutting down.
I20260619 09:03:22.136255 13692 raft_consensus.cc:2258] T 1b668d06ca044da2ba55145eb2ca6115 P 5712911e376c4b78b0d6c49a9495c911 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 09:03:22.136668 13692 ts_tablet_manager.cc:1905] T 1b668d06ca044da2ba55145eb2ca6115 P 5712911e376c4b78b0d6c49a9495c911: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 09:03:22.134785 13691 tablet_replica.cc:330] T 1b668d06ca044da2ba55145eb2ca6115 P 8bf3e72f7bc44976a8b60b1e5473c061: stopping tablet replica
I20260619 09:03:22.136948 13691 raft_consensus.cc:2229] T 1b668d06ca044da2ba55145eb2ca6115 P 8bf3e72f7bc44976a8b60b1e5473c061 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 09:03:22.137112 13691 raft_consensus.cc:2258] T 1b668d06ca044da2ba55145eb2ca6115 P 8bf3e72f7bc44976a8b60b1e5473c061 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 09:03:22.137550 13691 ts_tablet_manager.cc:1905] T 1b668d06ca044da2ba55145eb2ca6115 P 8bf3e72f7bc44976a8b60b1e5473c061: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 09:03:22.138123 13692 ts_tablet_manager.cc:1918] T 1b668d06ca044da2ba55145eb2ca6115 P 5712911e376c4b78b0d6c49a9495c911: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 09:03:22.138188 13692 log.cc:1196] T 1b668d06ca044da2ba55145eb2ca6115 P 5712911e376c4b78b0d6c49a9495c911: Deleting WAL directory at /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859658134510-392-0/minicluster-data/ts-1/wal/wals/1b668d06ca044da2ba55145eb2ca6115
I20260619 09:03:22.138408 13692 ts_tablet_manager.cc:1939] T 1b668d06ca044da2ba55145eb2ca6115 P 5712911e376c4b78b0d6c49a9495c911: Deleting consensus metadata
I20260619 09:03:22.138854 13145 catalog_manager.cc:4841] TS 5712911e376c4b78b0d6c49a9495c911 (127.0.98.2:33713): tablet 1b668d06ca044da2ba55145eb2ca6115 (table test-table [id=6c529532bb374f40ac50149f9fba1b50]) successfully deleted
I20260619 09:03:22.139341 13691 ts_tablet_manager.cc:1918] T 1b668d06ca044da2ba55145eb2ca6115 P 8bf3e72f7bc44976a8b60b1e5473c061: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 09:03:22.139408 13691 log.cc:1196] T 1b668d06ca044da2ba55145eb2ca6115 P 8bf3e72f7bc44976a8b60b1e5473c061: Deleting WAL directory at /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781859658134510-392-0/minicluster-data/ts-2/wal/wals/1b668d06ca044da2ba55145eb2ca6115
I20260619 09:03:22.139616 13691 ts_tablet_manager.cc:1939] T 1b668d06ca044da2ba55145eb2ca6115 P 8bf3e72f7bc44976a8b60b1e5473c061: Deleting consensus metadata
I20260619 09:03:22.139990 13145 catalog_manager.cc:4841] TS 8bf3e72f7bc44976a8b60b1e5473c061 (127.0.98.3:39527): tablet 1b668d06ca044da2ba55145eb2ca6115 (table test-table [id=6c529532bb374f40ac50149f9fba1b50]) successfully deleted
W20260619 09:03:22.140333 13144 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:40147 (error 108)
I20260619 09:03:22.140570 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ez5NB/build/release/bin/kudu with pid 13361
I20260619 09:03:22.141817 13654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:03:22.141185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40147 (local address 127.0.0.1:49764)
0619 09:03:22.141755 (+ 570us) negotiation.cc:107] Waiting for socket to connect
0619 09:03:22.141790 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":553}
W20260619 09:03:22.141953 13144 catalog_manager.cc:4584] TS 412d20e190f64b59b233e4b7da26de6b (127.0.98.1:40147): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1b668d06ca044da2ba55145eb2ca6115: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40147: connect: Connection refused (error 111)
I20260619 09:03:22.145208 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ez5NB/build/release/bin/kudu with pid 13494
I20260619 09:03:22.149511 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ez5NB/build/release/bin/kudu with pid 13126
I20260619 09:03:22.156374 392 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260619 09:03:22.490423 392 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260619 09:03:22.490513 392 mini_ranger.cc:66] Stopping Ranger...
I20260619 09:03:22.824515 392 mini_ranger.cc:68] Stopped Ranger
2026-06-19 09:03:22.824 UTC [12193] LOG: received smart shutdown request
2026-06-19 09:03:22.828 UTC [12193] LOG: background worker "logical replication launcher" (PID 12202) exited with exit code 1
2026-06-19 09:03:22.828 UTC [12197] LOG: shutting down
2026-06-19 09:03:22.977 UTC [12193] LOG: database system is shut down
2026-06-19T09:03:22Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (45227 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781859658134510-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 09:03:23 dist-test-slave-mtdr krb5kdc[13702](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 09:03:23 dist-test-slave-mtdr krb5kdc[13702](info): set up 2 sockets
Jun 19 09:03:23 dist-test-slave-mtdr krb5kdc[13702](info): commencing operation
krb5kdc: starting...
W20260619 09:03:24.010478 392 mini_kdc.cc:121] Time spent starting KDC: real 0.926s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781859658134510-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781859658134510-392-0/krb5kdc/test-user.keytab.
Jun 19 09:03:24 dist-test-slave-mtdr krb5kdc[13702](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859804, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T09:03:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T09:03:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781859658134510-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781859658134510-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 09:03:24.116937 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ez5NB/build/release/bin/kudu
/tmp/dist-test-task4ez5NB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781859658134510-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781859658134510-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781859658134510-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781859658134510-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39819
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781859658134510-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33077
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39819,127.0.98.61:40155,127.0.98.60:34781
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781859658134510-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781859658134510-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781859658134510-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 09:03:24.193409 13715 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 09:03:24.193604 13715 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 09:03:24.193625 13715 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 09:03:24.194782 13715 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 09:03:24.194852 13715 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 09:03:24.194871 13715 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 09:03:24.194888 13715 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 09:03:24.194912 13715 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 09:03:24.196370 13715 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33077
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781859658134510-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781859658134510-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39819,127.0.98.61:40155,127.0.98.60:34781
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781859658134510-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39819
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781859658134510-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13715
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781859658134510-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 09:03:24.196566 13715 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 09:03:24.196729 13715 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 09:03:24.201159 13721 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:03:24.201274 13715 server_base.cc:1034] running on GCE node
W20260619 09:03:24.201150 13723 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:03:24.201150 13720 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:03:24.201634 13715 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 09:03:24.201902 13715 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 09:03:24.203053 13715 hybrid_clock.cc:648] HybridClock initialized: now 1781859804203040 us; error 26 us; skew 500 ppm
Jun 19 09:03:24 dist-test-slave-mtdr krb5kdc[13702](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859804, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 09:03:24.208432 13715 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 09:03:24.208842 13715 webserver.cc:460] Webserver started at http://127.0.98.62:39743/ using document root <none> and password file <none>
I20260619 09:03:24.209041 13715 fs_manager.cc:359] Metadata directory not provided
I20260619 09:03:24.209092 13715 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 09:03:24.209213 13715 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 09:03:24.210225 13715 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781859658134510-392-0/minicluster-data/master-0/data/instance:
uuid: "a7f2f0b561194045915d9bff35132b85"
format_stamp: "Formatted at 2026-06-19 09:03:24 on dist-test-slave-mtdr"
server_key: "e251eafc394998d28476044d116814b6"
server_key_iv: "d8aee4eabd2648db6d6ec3aa838796d9"
server_key_version: "encryptionkey@0"
I20260619 09:03:24.210546 13715 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781859658134510-392-0/minicluster-data/master-0/wal/instance:
uuid: "a7f2f0b561194045915d9bff35132b85"
format_stamp: "Formatted at 2026-06-19 09:03:24 on dist-test-slave-mtdr"
server_key: "e251eafc394998d28476044d116814b6"
server_key_iv: "d8aee4eabd2648db6d6ec3aa838796d9"
server_key_version: "encryptionkey@0"
I20260619 09:03:24.211795 13715 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 09:03:24.212467 13730 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 09:03:24.212626 13715 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 09:03:24.212703 13715 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781859658134510-392-0/minicluster-data/master-0/data,/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781859658134510-392-0/minicluster-data/master-0/wal
uuid: "a7f2f0b561194045915d9bff35132b85"
format_stamp: "Formatted at 2026-06-19 09:03:24 on dist-test-slave-mtdr"
server_key: "e251eafc394998d28476044d116814b6"
server_key_iv: "d8aee4eabd2648db6d6ec3aa838796d9"
server_key_version: "encryptionkey@0"
I20260619 09:03:24.212766 13715 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781859658134510-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781859658134510-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781859658134510-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 09:03:24.220192 13715 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 09:03:24.221985 13715 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 09:03:24.222105 13715 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 09:03:24.225852 13715 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39819
I20260619 09:03:24.225893 13782 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39819 every 8 connection(s)
I20260619 09:03:24.226178 13715 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781859658134510-392-0/minicluster-data/master-0/data/info.pb
I20260619 09:03:24.227007 13783 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 09:03:24.228410 13783 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39819 } has no permanent_uuid. Determining permanent_uuid...
I20260619 09:03:24.230686 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ez5NB/build/release/bin/kudu as pid 13715
I20260619 09:03:24.230788 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781859658134510-392-0/minicluster-data/master-0/wal/instance
I20260619 09:03:24.231042 392 external_mini_cluster.cc:1428] Setting key c87bc0d61363b2f8ae5c2e673b423e9c
Jun 19 09:03:24 dist-test-slave-mtdr krb5kdc[13702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859804, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 09:03:24.237596 13785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:03:24.229041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50560 (local address 127.0.98.62:39819)
0619 09:03:24.229194 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0619 09:03:24.229201 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 09:03:24.229328 (+ 127us) server_negotiation.cc:407] Connection header received
0619 09:03:24.229791 (+ 463us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:03:24.229792 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:03:24.229808 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:03:24.229890 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:03:24.230335 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:03:24.231523 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:03:24.232346 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:03:24.232809 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:03:24.232929 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:03:24.234782 (+ 1853us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:03:24.234793 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:03:24.234793 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:03:24.234799 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:03:24.236228 (+ 1429us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:03:24.236637 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:03:24.236639 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:03:24.236640 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:03:24.236706 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:03:24.236920 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:03:24.236921 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:03:24.236921 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:03:24.237097 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:03:24.237156 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0619 09:03:24.237358 (+ 202us) server_negotiation.cc:299] Negotiation successful
0619 09:03:24.237451 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":77,"threads_started":1}
I20260619 09:03:24.237669 13786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:03:24.229038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39819 (local address 127.0.0.1:50560)
0619 09:03:24.229215 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0619 09:03:24.229252 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0619 09:03:24.229682 (+ 430us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:03:24.229899 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:03:24.229900 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:03:24.230124 (+ 224us) client_negotiation.cc:189] Negotiated authn=SASL
0619 09:03:24.230269 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:03:24.230273 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:03:24.231607 (+ 1334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:03:24.231608 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:03:24.232271 (+ 663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:03:24.232272 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:03:24.232883 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:03:24.232884 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:03:24.232932 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:03:24.233393 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 09:03:24.233400 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0619 09:03:24.234687 (+ 1287us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 09:03:24.236309 (+ 1622us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:03:24.236317 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:03:24.236319 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:03:24.236568 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:03:24.236760 (+ 192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:03:24.236760 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:03:24.236761 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:03:24.236864 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:03:24.237160 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 09:03:24.237166 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 09:03:24.237239 (+ 73us) client_negotiation.cc:769] Sending connection context
0619 09:03:24.237345 (+ 106us) client_negotiation.cc:240] Negotiation successful
0619 09:03:24.237541 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"mutex_wait_us":56,"thread_start_us":91,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
I20260619 09:03:24.238336 13783 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40155 } has no permanent_uuid. Determining permanent_uuid...
I20260619 09:03:24.238631 13786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:03:24.238537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40155 (local address 127.0.0.1:54988)
0619 09:03:24.238577 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0619 09:03:24.238615 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260619 09:03:24.238746 13734 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40155: connect: Connection refused (error 111)
I20260619 09:03:24.239184 13786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:03:24.239101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40155 (local address 127.0.0.1:55002)
0619 09:03:24.239141 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0619 09:03:24.239167 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260619 09:03:24.239288 13783 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:40155: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40155: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781859658134510-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781859658134510-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260619 09:03:24.253659 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ez5NB/build/release/bin/kudu
/tmp/dist-test-task4ez5NB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781859658134510-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781859658134510-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781859658134510-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781859658134510-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:40155
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781859658134510-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33077
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39819,127.0.98.61:40155,127.0.98.60:34781
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781859658134510-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781859658134510-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781859658134510-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 09:03:24.288517 13783 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40155 } attempt: 1
I20260619 09:03:24.289014 13786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:03:24.288865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40155 (local address 127.0.0.1:55016)
0619 09:03:24.288948 (+ 83us) negotiation.cc:107] Waiting for socket to connect
0619 09:03:24.288984 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260619 09:03:24.289315 13786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:03:24.289237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40155 (local address 127.0.0.1:55018)
0619 09:03:24.289272 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0619 09:03:24.289296 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260619 09:03:24.289407 13783 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:40155: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40155: connect: Connection refused (error 111)
W20260619 09:03:24.331301 13790 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 09:03:24.331491 13790 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 09:03:24.331521 13790 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 09:03:24.332605 13790 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 09:03:24.332641 13790 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 09:03:24.332660 13790 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 09:03:24.332674 13790 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 09:03:24.332690 13790 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 09:03:24.334128 13790 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33077
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781859658134510-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781859658134510-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39819,127.0.98.61:40155,127.0.98.60:34781
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781859658134510-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:40155
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781859658134510-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13790
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781859658134510-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 09:03:24.334347 13790 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 09:03:24.334529 13790 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 09:03:24.339002 13798 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:03:24.339102 13795 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:03:24.339075 13790 server_base.cc:1034] running on GCE node
W20260619 09:03:24.339082 13796 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:03:24.339498 13790 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 09:03:24.339768 13790 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 09:03:24.340914 13790 hybrid_clock.cc:648] HybridClock initialized: now 1781859804340903 us; error 30 us; skew 500 ppm
Jun 19 09:03:24 dist-test-slave-mtdr krb5kdc[13702](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859804, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 09:03:24.343268 13790 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260619 09:03:24.343676 13790 webserver.cc:460] Webserver started at http://127.0.98.61:34701/ using document root <none> and password file <none>
I20260619 09:03:24.343883 13790 fs_manager.cc:359] Metadata directory not provided
I20260619 09:03:24.343930 13790 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 09:03:24.344046 13790 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 09:03:24.345031 13790 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781859658134510-392-0/minicluster-data/master-1/data/instance:
uuid: "248489a7c4e34189b31481a54c940491"
format_stamp: "Formatted at 2026-06-19 09:03:24 on dist-test-slave-mtdr"
server_key: "f0d66d16f6c83ffd75e91be7aace2797"
server_key_iv: "ee90b6ff47e53523b34d4fd4e4ba6c0b"
server_key_version: "encryptionkey@0"
I20260619 09:03:24.345348 13790 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781859658134510-392-0/minicluster-data/master-1/wal/instance:
uuid: "248489a7c4e34189b31481a54c940491"
format_stamp: "Formatted at 2026-06-19 09:03:24 on dist-test-slave-mtdr"
server_key: "f0d66d16f6c83ffd75e91be7aace2797"
server_key_iv: "ee90b6ff47e53523b34d4fd4e4ba6c0b"
server_key_version: "encryptionkey@0"
I20260619 09:03:24.346526 13790 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 09:03:24.347267 13805 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 09:03:24.347434 13790 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260619 09:03:24.347517 13790 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781859658134510-392-0/minicluster-data/master-1/data,/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781859658134510-392-0/minicluster-data/master-1/wal
uuid: "248489a7c4e34189b31481a54c940491"
format_stamp: "Formatted at 2026-06-19 09:03:24 on dist-test-slave-mtdr"
server_key: "f0d66d16f6c83ffd75e91be7aace2797"
server_key_iv: "ee90b6ff47e53523b34d4fd4e4ba6c0b"
server_key_version: "encryptionkey@0"
I20260619 09:03:24.347582 13790 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781859658134510-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781859658134510-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781859658134510-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 09:03:24.357549 13783 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40155 } attempt: 2
I20260619 09:03:24.357904 13786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:03:24.357782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40155 (local address 127.0.0.1:55028)
0619 09:03:24.357836 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0619 09:03:24.357871 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260619 09:03:24.358249 13786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:03:24.358167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40155 (local address 127.0.0.1:55042)
0619 09:03:24.358208 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0619 09:03:24.358232 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260619 09:03:24.358353 13783 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:40155: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40155: connect: Connection refused (error 111)
I20260619 09:03:24.364523 13790 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 09:03:24.366235 13790 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 09:03:24.366357 13790 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 09:03:24.370213 13790 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:40155
I20260619 09:03:24.370244 13857 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:40155 every 8 connection(s)
I20260619 09:03:24.370550 13790 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781859658134510-392-0/minicluster-data/master-1/data/info.pb
I20260619 09:03:24.371368 13858 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 09:03:24.372841 13858 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39819 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 09:03:24 dist-test-slave-mtdr krb5kdc[13702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859804, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 09:03:24.377914 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ez5NB/build/release/bin/kudu as pid 13790
I20260619 09:03:24.378016 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781859658134510-392-0/minicluster-data/master-1/wal/instance
I20260619 09:03:24.378206 392 external_mini_cluster.cc:1428] Setting key dafc473cdce215d75fc331cd80e40dbd
I20260619 09:03:24.380718 13785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:03:24.373465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50576 (local address 127.0.98.62:39819)
0619 09:03:24.373534 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0619 09:03:24.373537 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 09:03:24.373710 (+ 173us) server_negotiation.cc:407] Connection header received
0619 09:03:24.374162 (+ 452us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:03:24.374163 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:03:24.374174 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:03:24.374224 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:03:24.374746 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:03:24.375252 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:03:24.376151 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:03:24.376337 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:03:24.376382 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:03:24.378268 (+ 1886us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:03:24.378269 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:03:24.378269 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:03:24.378277 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:03:24.379480 (+ 1203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:03:24.379833 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:03:24.379834 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:03:24.379834 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:03:24.379884 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:03:24.380130 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:03:24.380131 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:03:24.380132 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:03:24.380235 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:03:24.380290 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0619 09:03:24.380590 (+ 300us) server_negotiation.cc:299] Negotiation successful
0619 09:03:24.380656 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260619 09:03:24.380908 13860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:03:24.373447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39819 (local address 127.0.0.1:50576)
0619 09:03:24.373603 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0619 09:03:24.373637 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0619 09:03:24.374088 (+ 451us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:03:24.374275 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:03:24.374278 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:03:24.374506 (+ 228us) client_negotiation.cc:189] Negotiated authn=SASL
0619 09:03:24.374688 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:03:24.374692 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:03:24.375322 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:03:24.375323 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:03:24.376092 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:03:24.376094 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:03:24.376387 (+ 293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:03:24.376388 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:03:24.376478 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:03:24.376844 (+ 366us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 09:03:24.376851 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0619 09:03:24.378186 (+ 1335us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 09:03:24.379558 (+ 1372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:03:24.379565 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:03:24.379568 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:03:24.379766 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:03:24.379968 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:03:24.379969 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:03:24.379969 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:03:24.380075 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:03:24.380289 (+ 214us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 09:03:24.380296 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 09:03:24.380454 (+ 158us) client_negotiation.cc:769] Sending connection context
0619 09:03:24.380557 (+ 103us) client_negotiation.cc:240] Negotiation successful
0619 09:03:24.380704 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":77,"threads_started":1}
I20260619 09:03:24.381408 13858 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40155 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Jun 19 09:03:24 dist-test-slave-mtdr krb5kdc[13702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859804, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
I20260619 09:03:24.388576 13860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:03:24.381650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40155 (local address 127.0.0.1:55048)
0619 09:03:24.381725 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0619 09:03:24.381734 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 09:03:24.381774 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:03:24.382002 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:03:24.382002 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:03:24.382100 (+ 98us) client_negotiation.cc:189] Negotiated authn=SASL
0619 09:03:24.382144 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:03:24.382145 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:03:24.383344 (+ 1199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:03:24.383344 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:03:24.383814 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:03:24.383815 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:03:24.384398 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:03:24.384399 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:03:24.384448 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:03:24.384760 (+ 312us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 09:03:24.384764 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 09:03:24.386034 (+ 1270us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 09:03:24.387599 (+ 1565us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:03:24.387600 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:03:24.387600 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:03:24.387762 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:03:24.387968 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:03:24.387969 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:03:24.387969 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:03:24.388029 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:03:24.388248 (+ 219us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 09:03:24.388248 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 09:03:24.388290 (+ 42us) client_negotiation.cc:769] Sending connection context
0619 09:03:24.388357 (+ 67us) client_negotiation.cc:240] Negotiation successful
0619 09:03:24.388448 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61}
I20260619 09:03:24.388589 13862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:03:24.381695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55048 (local address 127.0.98.61:40155)
0619 09:03:24.381854 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0619 09:03:24.381859 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 09:03:24.381883 (+ 24us) server_negotiation.cc:407] Connection header received
0619 09:03:24.381905 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:03:24.381907 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:03:24.381920 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:03:24.381988 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:03:24.382200 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:03:24.383268 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:03:24.383882 (+ 614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:03:24.384322 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:03:24.384395 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:03:24.386116 (+ 1721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:03:24.386122 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:03:24.386123 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:03:24.386128 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:03:24.387528 (+ 1400us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:03:24.387833 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:03:24.387837 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:03:24.387837 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:03:24.387888 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:03:24.388084 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:03:24.388085 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:03:24.388085 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:03:24.388198 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:03:24.388252 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0619 09:03:24.388378 (+ 126us) server_negotiation.cc:299] Negotiation successful
0619 09:03:24.388491 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":53,"server-negotiator.queue_time_us":130,"thread_start_us":90,"threads_started":1}
I20260619 09:03:24.389199 13858 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34781 } has no permanent_uuid. Determining permanent_uuid...
I20260619 09:03:24.389477 13860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:03:24.389385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34781 (local address 127.0.0.1:37554)
0619 09:03:24.389413 (+ 28us) negotiation.cc:107] Waiting for socket to connect
0619 09:03:24.389456 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":15}
W20260619 09:03:24.389591 13806 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34781: connect: Connection refused (error 111)
I20260619 09:03:24.389972 13860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:03:24.389866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34781 (local address 127.0.0.1:37568)
0619 09:03:24.389905 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0619 09:03:24.389956 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260619 09:03:24.390060 13858 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:34781: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34781: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781859658134510-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781859658134510-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260619 09:03:24.401518 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ez5NB/build/release/bin/kudu
/tmp/dist-test-task4ez5NB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781859658134510-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781859658134510-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781859658134510-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781859658134510-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:34781
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781859658134510-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33077
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39819,127.0.98.61:40155,127.0.98.60:34781
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781859658134510-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781859658134510-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781859658134510-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 09:03:24.439267 13858 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34781 } attempt: 1
I20260619 09:03:24.439730 13860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:03:24.439596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34781 (local address 127.0.0.1:37574)
0619 09:03:24.439654 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0619 09:03:24.439690 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260619 09:03:24.440201 13860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:03:24.440104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34781 (local address 127.0.0.1:37578)
0619 09:03:24.440131 (+ 27us) negotiation.cc:107] Waiting for socket to connect
0619 09:03:24.440157 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":13}
W20260619 09:03:24.440343 13858 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:34781: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34781: connect: Connection refused (error 111)
I20260619 09:03:24.449489 13783 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40155 } attempt: 3
Jun 19 09:03:24 dist-test-slave-mtdr krb5kdc[13702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859804, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260619 09:03:24.455880 13862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:03:24.449784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55064 (local address 127.0.98.61:40155)
0619 09:03:24.449842 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0619 09:03:24.449844 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:03:24.449855 (+ 11us) server_negotiation.cc:407] Connection header received
0619 09:03:24.449941 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:03:24.449942 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:03:24.449952 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:03:24.450000 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:03:24.450326 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:03:24.450933 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:03:24.451585 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:03:24.451793 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:03:24.451847 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:03:24.453728 (+ 1881us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:03:24.453729 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:03:24.453729 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:03:24.453736 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:03:24.454944 (+ 1208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:03:24.455222 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:03:24.455223 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:03:24.455223 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:03:24.455271 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:03:24.455436 (+ 165us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:03:24.455437 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:03:24.455437 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:03:24.455537 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:03:24.455591 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0619 09:03:24.455722 (+ 131us) server_negotiation.cc:299] Negotiation successful
0619 09:03:24.455789 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260619 09:03:24.455892 13786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:03:24.449766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40155 (local address 127.0.0.1:55064)
0619 09:03:24.449805 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0619 09:03:24.449815 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 09:03:24.449864 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:03:24.450011 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:03:24.450012 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:03:24.450160 (+ 148us) client_negotiation.cc:189] Negotiated authn=SASL
0619 09:03:24.450225 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:03:24.450227 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:03:24.451026 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:03:24.451028 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:03:24.451517 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:03:24.451519 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:03:24.451851 (+ 332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:03:24.451852 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:03:24.451923 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:03:24.452361 (+ 438us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 09:03:24.452366 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 09:03:24.453652 (+ 1286us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 09:03:24.455009 (+ 1357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:03:24.455010 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:03:24.455010 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:03:24.455164 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:03:24.455327 (+ 163us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:03:24.455327 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:03:24.455328 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:03:24.455384 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:03:24.455589 (+ 205us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 09:03:24.455590 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 09:03:24.455636 (+ 46us) client_negotiation.cc:769] Sending connection context
0619 09:03:24.455695 (+ 59us) client_negotiation.cc:240] Negotiation successful
0619 09:03:24.455798 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18}
I20260619 09:03:24.456250 13783 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34781 } has no permanent_uuid. Determining permanent_uuid...
I20260619 09:03:24.456552 13786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:03:24.456463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34781 (local address 127.0.0.1:37584)
0619 09:03:24.456505 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0619 09:03:24.456529 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260619 09:03:24.456795 13786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:03:24.456720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34781 (local address 127.0.0.1:37592)
0619 09:03:24.456755 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0619 09:03:24.456776 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260619 09:03:24.456862 13783 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:34781: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34781: connect: Connection refused (error 111)
W20260619 09:03:24.480134 13865 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 09:03:24.480314 13865 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 09:03:24.480351 13865 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 09:03:24.481429 13865 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 09:03:24.481467 13865 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 09:03:24.481485 13865 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 09:03:24.481500 13865 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 09:03:24.481516 13865 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 09:03:24.482971 13865 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33077
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781859658134510-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781859658134510-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39819,127.0.98.61:40155,127.0.98.60:34781
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781859658134510-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:34781
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781859658134510-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13865
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781859658134510-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 09:03:24.483186 13865 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 09:03:24.483372 13865 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 09:03:24.487695 13873 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:03:24.487733 13870 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:03:24.487751 13871 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:03:24.487841 13865 server_base.cc:1034] running on GCE node
I20260619 09:03:24.487975 13783 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34781 } attempt: 1
I20260619 09:03:24.488116 13865 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 09:03:24.488307 13786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:03:24.488201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34781 (local address 127.0.0.1:37594)
0619 09:03:24.488250 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0619 09:03:24.488276 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260619 09:03:24.488408 13865 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 09:03:24.488691 13786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:03:24.488585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34781 (local address 127.0.0.1:37608)
0619 09:03:24.488631 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0619 09:03:24.488660 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260619 09:03:24.488780 13783 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:34781: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34781: connect: Connection refused (error 111)
I20260619 09:03:24.489559 13865 hybrid_clock.cc:648] HybridClock initialized: now 1781859804489548 us; error 24 us; skew 500 ppm
Jun 19 09:03:24 dist-test-slave-mtdr krb5kdc[13702](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859804, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 09:03:24.491935 13865 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260619 09:03:24.492347 13865 webserver.cc:460] Webserver started at http://127.0.98.60:44265/ using document root <none> and password file <none>
I20260619 09:03:24.492513 13865 fs_manager.cc:359] Metadata directory not provided
I20260619 09:03:24.492553 13865 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 09:03:24.492645 13865 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 09:03:24.493606 13865 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781859658134510-392-0/minicluster-data/master-2/data/instance:
uuid: "60921ac3ded04ef5ad071bee374b90bd"
format_stamp: "Formatted at 2026-06-19 09:03:24 on dist-test-slave-mtdr"
server_key: "f3fad788d961f29e2d8f1a487344222d"
server_key_iv: "113dbeeb0a8583a808234d33180aef0b"
server_key_version: "encryptionkey@0"
I20260619 09:03:24.493906 13865 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781859658134510-392-0/minicluster-data/master-2/wal/instance:
uuid: "60921ac3ded04ef5ad071bee374b90bd"
format_stamp: "Formatted at 2026-06-19 09:03:24 on dist-test-slave-mtdr"
server_key: "f3fad788d961f29e2d8f1a487344222d"
server_key_iv: "113dbeeb0a8583a808234d33180aef0b"
server_key_version: "encryptionkey@0"
I20260619 09:03:24.495143 13865 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 09:03:24.495833 13880 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 09:03:24.495957 13865 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 09:03:24.496034 13865 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781859658134510-392-0/minicluster-data/master-2/data,/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781859658134510-392-0/minicluster-data/master-2/wal
uuid: "60921ac3ded04ef5ad071bee374b90bd"
format_stamp: "Formatted at 2026-06-19 09:03:24 on dist-test-slave-mtdr"
server_key: "f3fad788d961f29e2d8f1a487344222d"
server_key_iv: "113dbeeb0a8583a808234d33180aef0b"
server_key_version: "encryptionkey@0"
I20260619 09:03:24.496102 13865 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781859658134510-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781859658134510-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781859658134510-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 09:03:24.508510 13858 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34781 } attempt: 2
I20260619 09:03:24.508909 13860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:03:24.508772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34781 (local address 127.0.0.1:37614)
0619 09:03:24.508846 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0619 09:03:24.508881 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260619 09:03:24.509227 13860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:03:24.509150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34781 (local address 127.0.0.1:37622)
0619 09:03:24.509184 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0619 09:03:24.509208 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260619 09:03:24.509315 13858 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:34781: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34781: connect: Connection refused (error 111)
I20260619 09:03:24.509682 13865 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 09:03:24.511755 13865 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 09:03:24.511868 13865 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 09:03:24.515897 13865 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:34781
I20260619 09:03:24.515914 13932 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:34781 every 8 connection(s)
I20260619 09:03:24.516299 13865 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781859658134510-392-0/minicluster-data/master-2/data/info.pb
I20260619 09:03:24.517071 13933 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 09:03:24.518472 13933 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39819 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 09:03:24 dist-test-slave-mtdr krb5kdc[13702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859804, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 09:03:24.525840 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ez5NB/build/release/bin/kudu as pid 13865
I20260619 09:03:24.525944 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781859658134510-392-0/minicluster-data/master-2/wal/instance
I20260619 09:03:24.526177 392 external_mini_cluster.cc:1428] Setting key d9d0fda2f34bd8b407a53062596e0807
I20260619 09:03:24.526475 13785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:03:24.519161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50584 (local address 127.0.98.62:39819)
0619 09:03:24.519210 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0619 09:03:24.519212 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:03:24.519392 (+ 180us) server_negotiation.cc:407] Connection header received
0619 09:03:24.519805 (+ 413us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:03:24.519806 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:03:24.519818 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:03:24.519874 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:03:24.520393 (+ 519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:03:24.520854 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:03:24.521848 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:03:24.522062 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:03:24.522114 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:03:24.524010 (+ 1896us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:03:24.524011 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:03:24.524011 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:03:24.524017 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:03:24.525105 (+ 1088us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:03:24.525491 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:03:24.525492 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:03:24.525492 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:03:24.525548 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:03:24.525799 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:03:24.525800 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:03:24.525800 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:03:24.525899 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:03:24.525962 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0619 09:03:24.526281 (+ 319us) server_negotiation.cc:299] Negotiation successful
0619 09:03:24.526381 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260619 09:03:24.526628 13935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:03:24.519097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39819 (local address 127.0.0.1:50584)
0619 09:03:24.519254 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0619 09:03:24.519291 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0619 09:03:24.519728 (+ 437us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:03:24.519924 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:03:24.519927 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:03:24.520146 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0619 09:03:24.520329 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:03:24.520334 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:03:24.520965 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:03:24.520967 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:03:24.521754 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:03:24.521756 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:03:24.522122 (+ 366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:03:24.522123 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:03:24.522217 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:03:24.522589 (+ 372us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 09:03:24.522596 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0619 09:03:24.523925 (+ 1329us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 09:03:24.525177 (+ 1252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:03:24.525182 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:03:24.525184 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:03:24.525396 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:03:24.525608 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:03:24.525608 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:03:24.525608 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:03:24.525734 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:03:24.525966 (+ 232us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 09:03:24.525972 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 09:03:24.526127 (+ 155us) client_negotiation.cc:769] Sending connection context
0619 09:03:24.526248 (+ 121us) client_negotiation.cc:240] Negotiation successful
0619 09:03:24.526398 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":73,"threads_started":1}
I20260619 09:03:24.527396 13933 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40155 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 09:03:24 dist-test-slave-mtdr krb5kdc[13702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859804, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Jun 19 09:03:24 dist-test-slave-mtdr krb5kdc[13702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859804, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260619 09:03:24.534969 13862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:03:24.527743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55080 (local address 127.0.98.61:40155)
0619 09:03:24.527815 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0619 09:03:24.527817 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:03:24.527873 (+ 56us) server_negotiation.cc:407] Connection header received
0619 09:03:24.527952 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:03:24.527954 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:03:24.527965 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:03:24.528006 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:03:24.528312 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:03:24.529038 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:03:24.529947 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:03:24.530239 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:03:24.530302 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:03:24.532582 (+ 2280us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:03:24.532583 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:03:24.532583 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:03:24.532590 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:03:24.533919 (+ 1329us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:03:24.534206 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:03:24.534208 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:03:24.534208 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:03:24.534263 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:03:24.534476 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:03:24.534477 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:03:24.534477 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:03:24.534587 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:03:24.534634 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0619 09:03:24.534792 (+ 158us) server_negotiation.cc:299] Negotiation successful
0619 09:03:24.534882 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260619 09:03:24.534974 13935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:03:24.527717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40155 (local address 127.0.0.1:55080)
0619 09:03:24.527805 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0619 09:03:24.527815 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 09:03:24.527865 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:03:24.528044 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:03:24.528045 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:03:24.528181 (+ 136us) client_negotiation.cc:189] Negotiated authn=SASL
0619 09:03:24.528242 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:03:24.528243 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:03:24.529138 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:03:24.529139 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:03:24.529864 (+ 725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:03:24.529866 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:03:24.530310 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:03:24.530311 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:03:24.530373 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:03:24.530737 (+ 364us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 09:03:24.530742 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 09:03:24.532082 (+ 1340us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 09:03:24.533997 (+ 1915us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:03:24.533997 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:03:24.533998 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:03:24.534142 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:03:24.534350 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:03:24.534351 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:03:24.534352 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:03:24.534412 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:03:24.534645 (+ 233us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 09:03:24.534645 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 09:03:24.534690 (+ 45us) client_negotiation.cc:769] Sending connection context
0619 09:03:24.534768 (+ 78us) client_negotiation.cc:240] Negotiation successful
0619 09:03:24.534876 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":73,"mutex_wait_us":29}
I20260619 09:03:24.535300 13933 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34781 } has no permanent_uuid. Determining permanent_uuid...
I20260619 09:03:24.535698 13937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:03:24.526853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37632 (local address 127.0.98.60:34781)
0619 09:03:24.527396 (+ 543us) server_negotiation.cc:206] Beginning negotiation
0619 09:03:24.527400 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 09:03:24.527421 (+ 21us) server_negotiation.cc:407] Connection header received
0619 09:03:24.527443 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:03:24.527445 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:03:24.527459 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:03:24.527572 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:03:24.528402 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:03:24.529537 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:03:24.530464 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:03:24.530761 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:03:24.530838 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:03:24.532955 (+ 2117us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:03:24.532963 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:03:24.532963 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:03:24.532970 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:03:24.534434 (+ 1464us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:03:24.534777 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:03:24.534780 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:03:24.534780 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:03:24.534852 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:03:24.535113 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:03:24.535114 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:03:24.535114 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:03:24.535290 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:03:24.535420 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0619 09:03:24.535538 (+ 118us) server_negotiation.cc:299] Negotiation successful
0619 09:03:24.535608 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":511,"thread_start_us":70,"threads_started":1}
Jun 19 09:03:24 dist-test-slave-mtdr krb5kdc[13702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859804, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Jun 19 09:03:24 dist-test-slave-mtdr krb5kdc[13702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859804, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 09:03:24.541893 13935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:03:24.535531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34781 (local address 127.0.0.1:37642)
0619 09:03:24.535658 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0619 09:03:24.535667 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 09:03:24.535704 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:03:24.535952 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:03:24.535953 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:03:24.536087 (+ 134us) client_negotiation.cc:189] Negotiated authn=SASL
0619 09:03:24.536113 (+ 26us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:03:24.536114 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:03:24.536852 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:03:24.536853 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:03:24.537395 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:03:24.537397 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:03:24.537770 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:03:24.537771 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:03:24.537823 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:03:24.538130 (+ 307us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 09:03:24.538134 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 09:03:24.539399 (+ 1265us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 09:03:24.540947 (+ 1548us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:03:24.540948 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:03:24.540948 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:03:24.541141 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:03:24.541303 (+ 162us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:03:24.541304 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:03:24.541304 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:03:24.541357 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:03:24.541548 (+ 191us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 09:03:24.541549 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 09:03:24.541607 (+ 58us) client_negotiation.cc:769] Sending connection context
0619 09:03:24.541680 (+ 73us) client_negotiation.cc:240] Negotiation successful
0619 09:03:24.541764 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114}
I20260619 09:03:24.541973 13937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:03:24.535558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37642 (local address 127.0.98.60:34781)
0619 09:03:24.535850 (+ 292us) server_negotiation.cc:206] Beginning negotiation
0619 09:03:24.535852 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:03:24.535859 (+ 7us) server_negotiation.cc:407] Connection header received
0619 09:03:24.535875 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:03:24.535876 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:03:24.535884 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:03:24.535936 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:03:24.536182 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:03:24.536781 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:03:24.537459 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:03:24.537708 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:03:24.537752 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:03:24.539478 (+ 1726us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:03:24.539479 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:03:24.539479 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:03:24.539485 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:03:24.540879 (+ 1394us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:03:24.541202 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:03:24.541203 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:03:24.541204 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:03:24.541252 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:03:24.541407 (+ 155us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:03:24.541408 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:03:24.541409 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:03:24.541499 (+ 90us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:03:24.541547 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0619 09:03:24.541696 (+ 149us) server_negotiation.cc:299] Negotiation successful
0619 09:03:24.541817 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":52,"server-negotiator.queue_time_us":280,"thread_start_us":73,"threads_started":1}
I20260619 09:03:24.543198 13785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:03:24.536463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50596 (local address 127.0.98.62:39819)
0619 09:03:24.536498 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0619 09:03:24.536500 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:03:24.536507 (+ 7us) server_negotiation.cc:407] Connection header received
0619 09:03:24.536547 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:03:24.536547 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:03:24.536555 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:03:24.536615 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:03:24.537070 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:03:24.537510 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:03:24.538270 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:03:24.538445 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:03:24.538492 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:03:24.540718 (+ 2226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:03:24.540719 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:03:24.540719 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:03:24.540726 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:03:24.542191 (+ 1465us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:03:24.542489 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:03:24.542490 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:03:24.542490 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:03:24.542550 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:03:24.542705 (+ 155us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:03:24.542706 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:03:24.542707 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:03:24.542809 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:03:24.543010 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0619 09:03:24.543045 (+ 35us) server_negotiation.cc:299] Negotiation successful
0619 09:03:24.543109 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260619 09:03:24.543558 13933 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 60921ac3ded04ef5ad071bee374b90bd: Bootstrap starting.
I20260619 09:03:24.544476 13933 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 60921ac3ded04ef5ad071bee374b90bd: Neither blocks nor log segments found. Creating new log.
I20260619 09:03:24.544754 13933 log.cc:824] T 00000000000000000000000000000000 P 60921ac3ded04ef5ad071bee374b90bd: Log is configured to *not* fsync() on all Append() calls
I20260619 09:03:24.545503 13933 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 60921ac3ded04ef5ad071bee374b90bd: No bootstrap required, opened a new log
I20260619 09:03:24.547412 13933 raft_consensus.cc:348] T 00000000000000000000000000000000 P 60921ac3ded04ef5ad071bee374b90bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7f2f0b561194045915d9bff35132b85" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39819 } } peers { permanent_uuid: "248489a7c4e34189b31481a54c940491" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40155 } } peers { permanent_uuid: "60921ac3ded04ef5ad071bee374b90bd" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34781 } }
I20260619 09:03:24.547549 13933 raft_consensus.cc:374] T 00000000000000000000000000000000 P 60921ac3ded04ef5ad071bee374b90bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 09:03:24.547582 13933 raft_consensus.cc:729] T 00000000000000000000000000000000 P 60921ac3ded04ef5ad071bee374b90bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60921ac3ded04ef5ad071bee374b90bd, State: Initialized, Role: FOLLOWER
I20260619 09:03:24.547698 13933 consensus_queue.cc:260] T 00000000000000000000000000000000 P 60921ac3ded04ef5ad071bee374b90bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7f2f0b561194045915d9bff35132b85" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39819 } } peers { permanent_uuid: "248489a7c4e34189b31481a54c940491" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40155 } } peers { permanent_uuid: "60921ac3ded04ef5ad071bee374b90bd" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34781 } }
Jun 19 09:03:24 dist-test-slave-mtdr krb5kdc[13702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859804, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260619 09:03:24.547930 13940 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60921ac3ded04ef5ad071bee374b90bd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7f2f0b561194045915d9bff35132b85" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39819 } } peers { permanent_uuid: "248489a7c4e34189b31481a54c940491" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40155 } } peers { permanent_uuid: "60921ac3ded04ef5ad071bee374b90bd" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34781 } } }
I20260619 09:03:24.547984 13933 sys_catalog.cc:564] T 00000000000000000000000000000000 P 60921ac3ded04ef5ad071bee374b90bd [sys.catalog]: configured and running, proceeding with master startup.
I20260619 09:03:24.548020 13940 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60921ac3ded04ef5ad071bee374b90bd [sys.catalog]: This master's current role is: FOLLOWER
W20260619 09:03:24.549659 13951 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 60921ac3ded04ef5ad071bee374b90bd: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 09:03:24.549757 13951 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 09:03:24.550938 13862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:03:24.543791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55082 (local address 127.0.98.61:40155)
0619 09:03:24.543831 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0619 09:03:24.543833 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:03:24.543840 (+ 7us) server_negotiation.cc:407] Connection header received
0619 09:03:24.543861 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:03:24.543862 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:03:24.543871 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:03:24.543945 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:03:24.544506 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:03:24.545035 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:03:24.545828 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:03:24.546011 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:03:24.546058 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:03:24.548356 (+ 2298us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:03:24.548357 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:03:24.548357 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:03:24.548364 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:03:24.549937 (+ 1573us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:03:24.550214 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:03:24.550215 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:03:24.550216 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:03:24.550268 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:03:24.550450 (+ 182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:03:24.550451 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:03:24.550451 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:03:24.550553 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:03:24.550720 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0619 09:03:24.550754 (+ 34us) server_negotiation.cc:299] Negotiation successful
0619 09:03:24.550853 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260619 09:03:24.552220 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 09:03:24.549909 13865 master_runner.cc:186] Error getting master registration for 127.0.98.62:39819: OK, instance_id { permanent_uuid: "a7f2f0b561194045915d9bff35132b85" instance_seqno: 1781859804224782 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 09:03:24.552529 13865 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 09:03:24.560638 13862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:03:24.553524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55090 (local address 127.0.98.61:40155)
0619 09:03:24.553557 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0619 09:03:24.553558 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 09:03:24.553803 (+ 245us) server_negotiation.cc:407] Connection header received
0619 09:03:24.553889 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:03:24.553890 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:03:24.553898 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:03:24.553925 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:03:24.554581 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:03:24.555300 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:03:24.556341 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:03:24.556624 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:03:24.556660 (+ 36us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:03:24.558092 (+ 1432us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:03:24.558093 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:03:24.558093 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:03:24.558100 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:03:24.559485 (+ 1385us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:03:24.559841 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:03:24.559842 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:03:24.559842 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:03:24.559908 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:03:24.560100 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:03:24.560101 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:03:24.560101 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:03:24.560221 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:03:24.560285 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0619 09:03:24.560484 (+ 199us) server_negotiation.cc:299] Negotiation successful
0619 09:03:24.560553 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260619 09:03:24.560988 13785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:03:24.553455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50604 (local address 127.0.98.62:39819)
0619 09:03:24.553492 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0619 09:03:24.553494 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:03:24.553667 (+ 173us) server_negotiation.cc:407] Connection header received
0619 09:03:24.553721 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:03:24.553722 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:03:24.553732 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:03:24.553786 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:03:24.554454 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:03:24.555188 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:03:24.556168 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:03:24.556463 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:03:24.556532 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:03:24.558090 (+ 1558us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:03:24.558091 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:03:24.558091 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:03:24.558099 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:03:24.559665 (+ 1566us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:03:24.560076 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:03:24.560077 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:03:24.560078 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:03:24.560131 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:03:24.560312 (+ 181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:03:24.560312 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:03:24.560313 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:03:24.560426 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:03:24.560525 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0619 09:03:24.560821 (+ 296us) server_negotiation.cc:299] Negotiation successful
0619 09:03:24.560903 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260619 09:03:24.561048 13937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:03:24.553757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37658 (local address 127.0.98.60:34781)
0619 09:03:24.553792 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0619 09:03:24.553794 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:03:24.553877 (+ 83us) server_negotiation.cc:407] Connection header received
0619 09:03:24.553961 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:03:24.553962 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:03:24.553970 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:03:24.553996 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:03:24.554592 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:03:24.555299 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:03:24.556342 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:03:24.556636 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:03:24.556668 (+ 32us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:03:24.558210 (+ 1542us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:03:24.558211 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:03:24.558212 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:03:24.558218 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:03:24.559774 (+ 1556us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:03:24.560169 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:03:24.560170 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:03:24.560170 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:03:24.560219 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:03:24.560402 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:03:24.560403 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:03:24.560404 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:03:24.560558 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:03:24.560635 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0619 09:03:24.560894 (+ 259us) server_negotiation.cc:299] Negotiation successful
0619 09:03:24.560966 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260619 09:03:24.563915 13783 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34781 } attempt: 2
Jun 19 09:03:24 dist-test-slave-mtdr krb5kdc[13702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859804, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260619 09:03:24.569977 13937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:03:24.564212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37670 (local address 127.0.98.60:34781)
0619 09:03:24.564246 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0619 09:03:24.564247 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 09:03:24.564255 (+ 8us) server_negotiation.cc:407] Connection header received
0619 09:03:24.564304 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:03:24.564304 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:03:24.564313 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:03:24.564366 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:03:24.564613 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:03:24.565017 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:03:24.565922 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:03:24.566260 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:03:24.566321 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:03:24.567955 (+ 1634us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:03:24.567956 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:03:24.567956 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:03:24.567962 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:03:24.569082 (+ 1120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:03:24.569361 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:03:24.569362 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:03:24.569362 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:03:24.569405 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:03:24.569563 (+ 158us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:03:24.569563 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:03:24.569564 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:03:24.569650 (+ 86us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:03:24.569702 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0619 09:03:24.569831 (+ 129us) server_negotiation.cc:299] Negotiation successful
0619 09:03:24.569891 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260619 09:03:24.569979 13786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:03:24.564145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34781 (local address 127.0.0.1:37670)
0619 09:03:24.564199 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0619 09:03:24.564210 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 09:03:24.564254 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:03:24.564367 (+ 113us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:03:24.564368 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:03:24.564519 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0619 09:03:24.564555 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:03:24.564557 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:03:24.565084 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:03:24.565085 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:03:24.565861 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:03:24.565863 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:03:24.566336 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:03:24.566337 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:03:24.566393 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:03:24.566743 (+ 350us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 09:03:24.566747 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 09:03:24.567883 (+ 1136us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 09:03:24.569155 (+ 1272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:03:24.569156 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:03:24.569156 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:03:24.569306 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:03:24.569454 (+ 148us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:03:24.569455 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:03:24.569455 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:03:24.569507 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:03:24.569702 (+ 195us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 09:03:24.569702 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 09:03:24.569743 (+ 41us) client_negotiation.cc:769] Sending connection context
0619 09:03:24.569802 (+ 59us) client_negotiation.cc:240] Negotiation successful
0619 09:03:24.569884 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260619 09:03:24.571269 13783 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a7f2f0b561194045915d9bff35132b85: Bootstrap starting.
I20260619 09:03:24.571890 13783 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a7f2f0b561194045915d9bff35132b85: Neither blocks nor log segments found. Creating new log.
I20260619 09:03:24.572170 13783 log.cc:824] T 00000000000000000000000000000000 P a7f2f0b561194045915d9bff35132b85: Log is configured to *not* fsync() on all Append() calls
I20260619 09:03:24.572925 13783 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a7f2f0b561194045915d9bff35132b85: No bootstrap required, opened a new log
I20260619 09:03:24.574882 13783 raft_consensus.cc:348] T 00000000000000000000000000000000 P a7f2f0b561194045915d9bff35132b85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7f2f0b561194045915d9bff35132b85" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39819 } } peers { permanent_uuid: "248489a7c4e34189b31481a54c940491" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40155 } } peers { permanent_uuid: "60921ac3ded04ef5ad071bee374b90bd" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34781 } }
I20260619 09:03:24.575017 13783 raft_consensus.cc:374] T 00000000000000000000000000000000 P a7f2f0b561194045915d9bff35132b85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 09:03:24.575059 13783 raft_consensus.cc:729] T 00000000000000000000000000000000 P a7f2f0b561194045915d9bff35132b85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7f2f0b561194045915d9bff35132b85, State: Initialized, Role: FOLLOWER
I20260619 09:03:24.575176 13783 consensus_queue.cc:260] T 00000000000000000000000000000000 P a7f2f0b561194045915d9bff35132b85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7f2f0b561194045915d9bff35132b85" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39819 } } peers { permanent_uuid: "248489a7c4e34189b31481a54c940491" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40155 } } peers { permanent_uuid: "60921ac3ded04ef5ad071bee374b90bd" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34781 } }
I20260619 09:03:24.575421 13961 sys_catalog.cc:455] T 00000000000000000000000000000000 P a7f2f0b561194045915d9bff35132b85 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7f2f0b561194045915d9bff35132b85" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39819 } } peers { permanent_uuid: "248489a7c4e34189b31481a54c940491" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40155 } } peers { permanent_uuid: "60921ac3ded04ef5ad071bee374b90bd" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34781 } } }
I20260619 09:03:24.575469 13783 sys_catalog.cc:564] T 00000000000000000000000000000000 P a7f2f0b561194045915d9bff35132b85 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 09:03:24.575495 13961 sys_catalog.cc:458] T 00000000000000000000000000000000 P a7f2f0b561194045915d9bff35132b85 [sys.catalog]: This master's current role is: FOLLOWER
W20260619 09:03:24.577183 13972 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a7f2f0b561194045915d9bff35132b85: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 09:03:24.577328 13972 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 09:03:24.577834 13715 master_runner.cc:186] Error getting master registration for 127.0.98.61:40155: OK, instance_id { permanent_uuid: "248489a7c4e34189b31481a54c940491" instance_seqno: 1781859804369187 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 09:03:24.580135 13715 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 09:03:24.600462 13858 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34781 } attempt: 3
Jun 19 09:03:24 dist-test-slave-mtdr krb5kdc[13702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859804, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260619 09:03:24.606570 13937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:03:24.600764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37684 (local address 127.0.98.60:34781)
0619 09:03:24.600808 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0619 09:03:24.600809 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 09:03:24.600817 (+ 8us) server_negotiation.cc:407] Connection header received
0619 09:03:24.600877 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:03:24.600878 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:03:24.600887 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:03:24.600930 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:03:24.601200 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:03:24.601664 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:03:24.602297 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:03:24.602472 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:03:24.602523 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:03:24.604241 (+ 1718us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:03:24.604242 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:03:24.604242 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:03:24.604250 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:03:24.605631 (+ 1381us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:03:24.605913 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:03:24.605914 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:03:24.605915 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:03:24.605965 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:03:24.606119 (+ 154us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:03:24.606120 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:03:24.606120 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:03:24.606225 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:03:24.606279 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0619 09:03:24.606403 (+ 124us) server_negotiation.cc:299] Negotiation successful
0619 09:03:24.606472 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260619 09:03:24.606580 13860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:03:24.600729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34781 (local address 127.0.0.1:37684)
0619 09:03:24.600777 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0619 09:03:24.600788 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 09:03:24.600828 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:03:24.600939 (+ 111us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:03:24.600940 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:03:24.601092 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0619 09:03:24.601135 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:03:24.601137 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:03:24.601734 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:03:24.601735 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:03:24.602227 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:03:24.602229 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:03:24.602526 (+ 297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:03:24.602527 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:03:24.602583 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:03:24.603005 (+ 422us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 09:03:24.603010 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 09:03:24.604148 (+ 1138us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 09:03:24.605705 (+ 1557us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:03:24.605707 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:03:24.605707 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:03:24.605857 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:03:24.606017 (+ 160us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:03:24.606018 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:03:24.606018 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:03:24.606072 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:03:24.606276 (+ 204us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 09:03:24.606277 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 09:03:24.606320 (+ 43us) client_negotiation.cc:769] Sending connection context
0619 09:03:24.606379 (+ 59us) client_negotiation.cc:240] Negotiation successful
0619 09:03:24.606472 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260619 09:03:24.607899 13858 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 248489a7c4e34189b31481a54c940491: Bootstrap starting.
I20260619 09:03:24.608496 13858 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 248489a7c4e34189b31481a54c940491: Neither blocks nor log segments found. Creating new log.
I20260619 09:03:24.608773 13858 log.cc:824] T 00000000000000000000000000000000 P 248489a7c4e34189b31481a54c940491: Log is configured to *not* fsync() on all Append() calls
I20260619 09:03:24.609465 13858 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 248489a7c4e34189b31481a54c940491: No bootstrap required, opened a new log
I20260619 09:03:24.610862 13858 raft_consensus.cc:348] T 00000000000000000000000000000000 P 248489a7c4e34189b31481a54c940491 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7f2f0b561194045915d9bff35132b85" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39819 } } peers { permanent_uuid: "248489a7c4e34189b31481a54c940491" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40155 } } peers { permanent_uuid: "60921ac3ded04ef5ad071bee374b90bd" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34781 } }
I20260619 09:03:24.610975 13858 raft_consensus.cc:374] T 00000000000000000000000000000000 P 248489a7c4e34189b31481a54c940491 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 09:03:24.611035 13858 raft_consensus.cc:729] T 00000000000000000000000000000000 P 248489a7c4e34189b31481a54c940491 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 248489a7c4e34189b31481a54c940491, State: Initialized, Role: FOLLOWER
I20260619 09:03:24.611171 13858 consensus_queue.cc:260] T 00000000000000000000000000000000 P 248489a7c4e34189b31481a54c940491 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7f2f0b561194045915d9bff35132b85" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39819 } } peers { permanent_uuid: "248489a7c4e34189b31481a54c940491" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40155 } } peers { permanent_uuid: "60921ac3ded04ef5ad071bee374b90bd" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34781 } }
I20260619 09:03:24.611407 13975 sys_catalog.cc:455] T 00000000000000000000000000000000 P 248489a7c4e34189b31481a54c940491 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7f2f0b561194045915d9bff35132b85" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39819 } } peers { permanent_uuid: "248489a7c4e34189b31481a54c940491" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40155 } } peers { permanent_uuid: "60921ac3ded04ef5ad071bee374b90bd" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34781 } } }
I20260619 09:03:24.611470 13858 sys_catalog.cc:564] T 00000000000000000000000000000000 P 248489a7c4e34189b31481a54c940491 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 09:03:24.611534 13975 sys_catalog.cc:458] T 00000000000000000000000000000000 P 248489a7c4e34189b31481a54c940491 [sys.catalog]: This master's current role is: FOLLOWER
W20260619 09:03:24.613175 13986 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 248489a7c4e34189b31481a54c940491: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 09:03:24.613272 13986 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 09:03:24.615954 13790 master_runner.cc:263] No leader master found from master 248489a7c4e34189b31481a54c940491
I20260619 09:03:24.615996 13790 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260619 09:03:24.616921 13811 tablet.cc:2369] T 00000000000000000000000000000000 P 248489a7c4e34189b31481a54c940491: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 09:03:24.722863 13736 tablet.cc:2369] T 00000000000000000000000000000000 P a7f2f0b561194045915d9bff35132b85: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 09:03:24.762462 13886 tablet.cc:2369] T 00000000000000000000000000000000 P 60921ac3ded04ef5ad071bee374b90bd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 09:03:24.881956 13961 raft_consensus.cc:482] T 00000000000000000000000000000000 P a7f2f0b561194045915d9bff35132b85 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 09:03:24.882090 13961 raft_consensus.cc:504] T 00000000000000000000000000000000 P a7f2f0b561194045915d9bff35132b85 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7f2f0b561194045915d9bff35132b85" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39819 } } peers { permanent_uuid: "248489a7c4e34189b31481a54c940491" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40155 } } peers { permanent_uuid: "60921ac3ded04ef5ad071bee374b90bd" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34781 } }
I20260619 09:03:24.882346 13961 leader_election.cc:288] T 00000000000000000000000000000000 P a7f2f0b561194045915d9bff35132b85 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 248489a7c4e34189b31481a54c940491 (127.0.98.61:40155), 60921ac3ded04ef5ad071bee374b90bd (127.0.98.60:34781)
I20260619 09:03:24.882638 13832 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a7f2f0b561194045915d9bff35132b85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "248489a7c4e34189b31481a54c940491" is_pre_election: true
I20260619 09:03:24.882637 13907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a7f2f0b561194045915d9bff35132b85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60921ac3ded04ef5ad071bee374b90bd" is_pre_election: true
I20260619 09:03:24.882774 13832 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 248489a7c4e34189b31481a54c940491 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a7f2f0b561194045915d9bff35132b85 in term 0.
I20260619 09:03:24.882784 13907 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 60921ac3ded04ef5ad071bee374b90bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a7f2f0b561194045915d9bff35132b85 in term 0.
I20260619 09:03:24.882974 13731 leader_election.cc:302] T 00000000000000000000000000000000 P a7f2f0b561194045915d9bff35132b85 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60921ac3ded04ef5ad071bee374b90bd, a7f2f0b561194045915d9bff35132b85; no voters:
I20260619 09:03:24.883126 13961 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a7f2f0b561194045915d9bff35132b85 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 09:03:24.883189 13961 raft_consensus.cc:482] T 00000000000000000000000000000000 P a7f2f0b561194045915d9bff35132b85 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 09:03:24.883205 13961 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a7f2f0b561194045915d9bff35132b85 [term 0 FOLLOWER]: Advancing to term 1
I20260619 09:03:24.883903 13961 raft_consensus.cc:504] T 00000000000000000000000000000000 P a7f2f0b561194045915d9bff35132b85 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7f2f0b561194045915d9bff35132b85" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39819 } } peers { permanent_uuid: "248489a7c4e34189b31481a54c940491" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40155 } } peers { permanent_uuid: "60921ac3ded04ef5ad071bee374b90bd" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34781 } }
I20260619 09:03:24.884054 13961 leader_election.cc:288] T 00000000000000000000000000000000 P a7f2f0b561194045915d9bff35132b85 [CANDIDATE]: Term 1 election: Requested vote from peers 248489a7c4e34189b31481a54c940491 (127.0.98.61:40155), 60921ac3ded04ef5ad071bee374b90bd (127.0.98.60:34781)
I20260619 09:03:24.884212 13832 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a7f2f0b561194045915d9bff35132b85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "248489a7c4e34189b31481a54c940491"
I20260619 09:03:24.884258 13907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a7f2f0b561194045915d9bff35132b85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60921ac3ded04ef5ad071bee374b90bd"
I20260619 09:03:24.884302 13832 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 248489a7c4e34189b31481a54c940491 [term 0 FOLLOWER]: Advancing to term 1
I20260619 09:03:24.884325 13907 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 60921ac3ded04ef5ad071bee374b90bd [term 0 FOLLOWER]: Advancing to term 1
I20260619 09:03:24.884984 13907 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 60921ac3ded04ef5ad071bee374b90bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a7f2f0b561194045915d9bff35132b85 in term 1.
I20260619 09:03:24.885058 13832 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 248489a7c4e34189b31481a54c940491 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a7f2f0b561194045915d9bff35132b85 in term 1.
I20260619 09:03:24.885142 13731 leader_election.cc:302] T 00000000000000000000000000000000 P a7f2f0b561194045915d9bff35132b85 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60921ac3ded04ef5ad071bee374b90bd, a7f2f0b561194045915d9bff35132b85; no voters:
I20260619 09:03:24.885223 13961 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a7f2f0b561194045915d9bff35132b85 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 09:03:24.885321 13961 raft_consensus.cc:686] T 00000000000000000000000000000000 P a7f2f0b561194045915d9bff35132b85 [term 1 LEADER]: Becoming Leader. State: Replica: a7f2f0b561194045915d9bff35132b85, State: Running, Role: LEADER
I20260619 09:03:24.885396 13961 consensus_queue.cc:237] T 00000000000000000000000000000000 P a7f2f0b561194045915d9bff35132b85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7f2f0b561194045915d9bff35132b85" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39819 } } peers { permanent_uuid: "248489a7c4e34189b31481a54c940491" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40155 } } peers { permanent_uuid: "60921ac3ded04ef5ad071bee374b90bd" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34781 } }
I20260619 09:03:24.885733 13988 sys_catalog.cc:455] T 00000000000000000000000000000000 P a7f2f0b561194045915d9bff35132b85 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a7f2f0b561194045915d9bff35132b85. Latest consensus state: current_term: 1 leader_uuid: "a7f2f0b561194045915d9bff35132b85" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7f2f0b561194045915d9bff35132b85" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39819 } } peers { permanent_uuid: "248489a7c4e34189b31481a54c940491" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40155 } } peers { permanent_uuid: "60921ac3ded04ef5ad071bee374b90bd" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34781 } } }
I20260619 09:03:24.885805 13988 sys_catalog.cc:458] T 00000000000000000000000000000000 P a7f2f0b561194045915d9bff35132b85 [sys.catalog]: This master's current role is: LEADER
I20260619 09:03:24.885993 13990 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 09:03:24.886164 13990 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 09:03:24.887507 13907 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 60921ac3ded04ef5ad071bee374b90bd [term 1 FOLLOWER]: Refusing update from remote peer a7f2f0b561194045915d9bff35132b85: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 09:03:24.887538 13832 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 248489a7c4e34189b31481a54c940491 [term 1 FOLLOWER]: Refusing update from remote peer a7f2f0b561194045915d9bff35132b85: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 09:03:24.887801 13961 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a7f2f0b561194045915d9bff35132b85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60921ac3ded04ef5ad071bee374b90bd" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 09:03:24.887909 13988 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a7f2f0b561194045915d9bff35132b85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "248489a7c4e34189b31481a54c940491" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 09:03:24.888932 13940 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60921ac3ded04ef5ad071bee374b90bd [sys.catalog]: SysCatalogTable state changed. Reason: New leader a7f2f0b561194045915d9bff35132b85. Latest consensus state: current_term: 1 leader_uuid: "a7f2f0b561194045915d9bff35132b85" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7f2f0b561194045915d9bff35132b85" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39819 } } peers { permanent_uuid: "248489a7c4e34189b31481a54c940491" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40155 } } peers { permanent_uuid: "60921ac3ded04ef5ad071bee374b90bd" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34781 } } }
I20260619 09:03:24.888949 13975 sys_catalog.cc:455] T 00000000000000000000000000000000 P 248489a7c4e34189b31481a54c940491 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a7f2f0b561194045915d9bff35132b85. Latest consensus state: current_term: 1 leader_uuid: "a7f2f0b561194045915d9bff35132b85" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7f2f0b561194045915d9bff35132b85" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39819 } } peers { permanent_uuid: "248489a7c4e34189b31481a54c940491" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40155 } } peers { permanent_uuid: "60921ac3ded04ef5ad071bee374b90bd" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34781 } } }
I20260619 09:03:24.889015 13940 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60921ac3ded04ef5ad071bee374b90bd [sys.catalog]: This master's current role is: FOLLOWER
I20260619 09:03:24.889022 13975 sys_catalog.cc:458] T 00000000000000000000000000000000 P 248489a7c4e34189b31481a54c940491 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 09:03:24.889873 13991 mvcc.cc:205] Tried to move back new op lower bound from 7298497760816947200 to 7298497760811024384. Current Snapshot: MvccSnapshot[applied={T|T < 7298497760816947200}]
I20260619 09:03:24.889943 13988 sys_catalog.cc:455] T 00000000000000000000000000000000 P a7f2f0b561194045915d9bff35132b85 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a7f2f0b561194045915d9bff35132b85" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7f2f0b561194045915d9bff35132b85" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39819 } } peers { permanent_uuid: "248489a7c4e34189b31481a54c940491" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40155 } } peers { permanent_uuid: "60921ac3ded04ef5ad071bee374b90bd" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34781 } } }
I20260619 09:03:24.890024 13988 sys_catalog.cc:458] T 00000000000000000000000000000000 P a7f2f0b561194045915d9bff35132b85 [sys.catalog]: This master's current role is: LEADER
I20260619 09:03:24.890167 13940 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60921ac3ded04ef5ad071bee374b90bd [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a7f2f0b561194045915d9bff35132b85" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7f2f0b561194045915d9bff35132b85" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39819 } } peers { permanent_uuid: "248489a7c4e34189b31481a54c940491" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40155 } } peers { permanent_uuid: "60921ac3ded04ef5ad071bee374b90bd" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34781 } } }
I20260619 09:03:24.890228 13940 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60921ac3ded04ef5ad071bee374b90bd [sys.catalog]: This master's current role is: FOLLOWER
I20260619 09:03:24.890359 13988 sys_catalog.cc:455] T 00000000000000000000000000000000 P a7f2f0b561194045915d9bff35132b85 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a7f2f0b561194045915d9bff35132b85" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7f2f0b561194045915d9bff35132b85" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39819 } } peers { permanent_uuid: "248489a7c4e34189b31481a54c940491" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40155 } } peers { permanent_uuid: "60921ac3ded04ef5ad071bee374b90bd" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34781 } } }
I20260619 09:03:24.890429 13988 sys_catalog.cc:458] T 00000000000000000000000000000000 P a7f2f0b561194045915d9bff35132b85 [sys.catalog]: This master's current role is: LEADER
I20260619 09:03:24.890570 13992 mvcc.cc:205] Tried to move back new op lower bound from 7298497760816947200 to 7298497760811024384. Current Snapshot: MvccSnapshot[applied={T|T < 7298497760816947200 or (T in {7298497760816947200})}]
I20260619 09:03:24.890702 13975 sys_catalog.cc:455] T 00000000000000000000000000000000 P 248489a7c4e34189b31481a54c940491 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a7f2f0b561194045915d9bff35132b85" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7f2f0b561194045915d9bff35132b85" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39819 } } peers { permanent_uuid: "248489a7c4e34189b31481a54c940491" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40155 } } peers { permanent_uuid: "60921ac3ded04ef5ad071bee374b90bd" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34781 } } }
I20260619 09:03:24.890772 13975 sys_catalog.cc:458] T 00000000000000000000000000000000 P 248489a7c4e34189b31481a54c940491 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 09:03:24.891013 13990 catalog_manager.cc:1347] Generated new cluster ID: 0768aeff67cb47c9aa7c3aed2a9a3646
I20260619 09:03:24.891076 13990 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 09:03:24.902097 13990 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 09:03:24.902717 13990 catalog_manager.cc:1504] Loading token signing keys...
I20260619 09:03:24.910185 13990 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a7f2f0b561194045915d9bff35132b85: Generated new TSK 0
I20260619 09:03:24.910445 13990 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 09:03:25.550220 13951 catalog_manager.cc:1259] Loaded cluster ID: 0768aeff67cb47c9aa7c3aed2a9a3646
I20260619 09:03:25.550316 13951 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 60921ac3ded04ef5ad071bee374b90bd: loading cluster ID for follower catalog manager: success
I20260619 09:03:25.551230 13951 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 60921ac3ded04ef5ad071bee374b90bd: acquiring CA information for follower catalog manager: success
I20260619 09:03:25.551512 13951 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 60921ac3ded04ef5ad071bee374b90bd: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 09:03:25.613729 13986 catalog_manager.cc:1259] Loaded cluster ID: 0768aeff67cb47c9aa7c3aed2a9a3646
I20260619 09:03:25.613803 13986 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 248489a7c4e34189b31481a54c940491: loading cluster ID for follower catalog manager: success
I20260619 09:03:25.614615 13986 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 248489a7c4e34189b31481a54c940491: acquiring CA information for follower catalog manager: success
I20260619 09:03:25.614941 13986 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 248489a7c4e34189b31481a54c940491: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 09:03:25.959280 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ez5NB/build/release/bin/kudu with pid 13715
I20260619 09:03:25.963760 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ez5NB/build/release/bin/kudu with pid 13790
I20260619 09:03:25.967722 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ez5NB/build/release/bin/kudu with pid 13865
2026-06-19T09:03:25Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (2900 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781859658134510-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 09:03:26 dist-test-slave-mtdr krb5kdc[14005](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 09:03:26 dist-test-slave-mtdr krb5kdc[14005](info): set up 2 sockets
Jun 19 09:03:26 dist-test-slave-mtdr krb5kdc[14005](info): commencing operation
krb5kdc: starting...
W20260619 09:03:26.914572 392 mini_kdc.cc:121] Time spent starting KDC: real 0.929s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781859658134510-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781859658134510-392-0/krb5kdc/test-user.keytab.
Jun 19 09:03:26 dist-test-slave-mtdr krb5kdc[14005](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859806, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T09:03:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T09:03:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781859658134510-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781859658134510-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 09:03:27.021020 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ez5NB/build/release/bin/kudu
/tmp/dist-test-task4ez5NB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781859658134510-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781859658134510-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781859658134510-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781859658134510-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40301
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781859658134510-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41183
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40301
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781859658134510-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781859658134510-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781859658134510-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 09:03:27.097710 14018 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 09:03:27.097892 14018 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 09:03:27.097913 14018 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 09:03:27.099058 14018 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 09:03:27.099097 14018 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 09:03:27.099109 14018 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 09:03:27.099131 14018 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 09:03:27.099155 14018 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 09:03:27.100579 14018 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41183
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781859658134510-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781859658134510-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40301
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781859658134510-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40301
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781859658134510-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14018
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781859658134510-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 09:03:27.100787 14018 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 09:03:27.101007 14018 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 09:03:27.105456 14024 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:03:27.105535 14018 server_base.cc:1034] running on GCE node
W20260619 09:03:27.105456 14026 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:03:27.105465 14023 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:03:27.105897 14018 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 09:03:27.106160 14018 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 09:03:27.107302 14018 hybrid_clock.cc:648] HybridClock initialized: now 1781859807107287 us; error 29 us; skew 500 ppm
Jun 19 09:03:27 dist-test-slave-mtdr krb5kdc[14005](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859807, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 09:03:27.109493 14018 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 09:03:27.109891 14018 webserver.cc:460] Webserver started at http://127.0.98.62:45125/ using document root <none> and password file <none>
I20260619 09:03:27.110085 14018 fs_manager.cc:359] Metadata directory not provided
I20260619 09:03:27.110131 14018 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 09:03:27.110273 14018 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 09:03:27.111260 14018 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781859658134510-392-0/minicluster-data/master-0/data/instance:
uuid: "28f6b889c2564926b5cb219e48192ef2"
format_stamp: "Formatted at 2026-06-19 09:03:27 on dist-test-slave-mtdr"
server_key: "b2fdfc8325c807d3fe97e82b54782d84"
server_key_iv: "b9d123486bf96e7974b0e4f57333c9d5"
server_key_version: "encryptionkey@0"
I20260619 09:03:27.111557 14018 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781859658134510-392-0/minicluster-data/master-0/wal/instance:
uuid: "28f6b889c2564926b5cb219e48192ef2"
format_stamp: "Formatted at 2026-06-19 09:03:27 on dist-test-slave-mtdr"
server_key: "b2fdfc8325c807d3fe97e82b54782d84"
server_key_iv: "b9d123486bf96e7974b0e4f57333c9d5"
server_key_version: "encryptionkey@0"
I20260619 09:03:27.112728 14018 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 09:03:27.113421 14033 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 09:03:27.113576 14018 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 09:03:27.113653 14018 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781859658134510-392-0/minicluster-data/master-0/data,/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781859658134510-392-0/minicluster-data/master-0/wal
uuid: "28f6b889c2564926b5cb219e48192ef2"
format_stamp: "Formatted at 2026-06-19 09:03:27 on dist-test-slave-mtdr"
server_key: "b2fdfc8325c807d3fe97e82b54782d84"
server_key_iv: "b9d123486bf96e7974b0e4f57333c9d5"
server_key_version: "encryptionkey@0"
I20260619 09:03:27.113710 14018 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781859658134510-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781859658134510-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781859658134510-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 09:03:27.132129 14018 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 09:03:27.133934 14018 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 09:03:27.134057 14018 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 09:03:27.138011 14018 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40301
I20260619 09:03:27.138043 14085 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40301 every 8 connection(s)
I20260619 09:03:27.138361 14018 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781859658134510-392-0/minicluster-data/master-0/data/info.pb
I20260619 09:03:27.139179 14086 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 09:03:27.141285 14086 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 28f6b889c2564926b5cb219e48192ef2: Bootstrap starting.
I20260619 09:03:27.141782 14086 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 28f6b889c2564926b5cb219e48192ef2: Neither blocks nor log segments found. Creating new log.
I20260619 09:03:27.142016 14086 log.cc:824] T 00000000000000000000000000000000 P 28f6b889c2564926b5cb219e48192ef2: Log is configured to *not* fsync() on all Append() calls
I20260619 09:03:27.142599 14086 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 28f6b889c2564926b5cb219e48192ef2: No bootstrap required, opened a new log
I20260619 09:03:27.144110 14086 raft_consensus.cc:348] T 00000000000000000000000000000000 P 28f6b889c2564926b5cb219e48192ef2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28f6b889c2564926b5cb219e48192ef2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40301 } }
I20260619 09:03:27.144243 14086 raft_consensus.cc:374] T 00000000000000000000000000000000 P 28f6b889c2564926b5cb219e48192ef2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 09:03:27.144276 14086 raft_consensus.cc:729] T 00000000000000000000000000000000 P 28f6b889c2564926b5cb219e48192ef2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28f6b889c2564926b5cb219e48192ef2, State: Initialized, Role: FOLLOWER
I20260619 09:03:27.144376 14086 consensus_queue.cc:260] T 00000000000000000000000000000000 P 28f6b889c2564926b5cb219e48192ef2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28f6b889c2564926b5cb219e48192ef2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40301 } }
I20260619 09:03:27.144421 14086 raft_consensus.cc:388] T 00000000000000000000000000000000 P 28f6b889c2564926b5cb219e48192ef2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 09:03:27.144445 14086 raft_consensus.cc:482] T 00000000000000000000000000000000 P 28f6b889c2564926b5cb219e48192ef2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 09:03:27.144474 14086 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 28f6b889c2564926b5cb219e48192ef2 [term 0 FOLLOWER]: Advancing to term 1
I20260619 09:03:27.144906 14086 raft_consensus.cc:504] T 00000000000000000000000000000000 P 28f6b889c2564926b5cb219e48192ef2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28f6b889c2564926b5cb219e48192ef2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40301 } }
I20260619 09:03:27.144986 14086 leader_election.cc:302] T 00000000000000000000000000000000 P 28f6b889c2564926b5cb219e48192ef2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 28f6b889c2564926b5cb219e48192ef2; no voters:
I20260619 09:03:27.145077 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ez5NB/build/release/bin/kudu as pid 14018
I20260619 09:03:27.145128 14086 leader_election.cc:288] T 00000000000000000000000000000000 P 28f6b889c2564926b5cb219e48192ef2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 09:03:27.145167 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781859658134510-392-0/minicluster-data/master-0/wal/instance
I20260619 09:03:27.145159 14089 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 28f6b889c2564926b5cb219e48192ef2 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 09:03:27.145336 14089 raft_consensus.cc:686] T 00000000000000000000000000000000 P 28f6b889c2564926b5cb219e48192ef2 [term 1 LEADER]: Becoming Leader. State: Replica: 28f6b889c2564926b5cb219e48192ef2, State: Running, Role: LEADER
I20260619 09:03:27.145344 392 external_mini_cluster.cc:1428] Setting key 98d7d6a90fe22df9d4bdc2017e5207ae
I20260619 09:03:27.145437 14086 sys_catalog.cc:564] T 00000000000000000000000000000000 P 28f6b889c2564926b5cb219e48192ef2 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 09:03:27.145466 14089 consensus_queue.cc:237] T 00000000000000000000000000000000 P 28f6b889c2564926b5cb219e48192ef2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28f6b889c2564926b5cb219e48192ef2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40301 } }
I20260619 09:03:27.145794 14089 sys_catalog.cc:455] T 00000000000000000000000000000000 P 28f6b889c2564926b5cb219e48192ef2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 28f6b889c2564926b5cb219e48192ef2. Latest consensus state: current_term: 1 leader_uuid: "28f6b889c2564926b5cb219e48192ef2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28f6b889c2564926b5cb219e48192ef2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40301 } } }
I20260619 09:03:27.145864 14089 sys_catalog.cc:458] T 00000000000000000000000000000000 P 28f6b889c2564926b5cb219e48192ef2 [sys.catalog]: This master's current role is: LEADER
I20260619 09:03:27.145952 14090 sys_catalog.cc:455] T 00000000000000000000000000000000 P 28f6b889c2564926b5cb219e48192ef2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "28f6b889c2564926b5cb219e48192ef2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28f6b889c2564926b5cb219e48192ef2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40301 } } }
I20260619 09:03:27.146036 14090 sys_catalog.cc:458] T 00000000000000000000000000000000 P 28f6b889c2564926b5cb219e48192ef2 [sys.catalog]: This master's current role is: LEADER
I20260619 09:03:27.146770 14095 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 09:03:27.147321 14095 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 09:03:27.148550 14095 catalog_manager.cc:1347] Generated new cluster ID: 5f180cf428124178b975d7bedcd70969
I20260619 09:03:27.148597 14095 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 19 09:03:27 dist-test-slave-mtdr krb5kdc[14005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859806, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 09:03:27.156634 14094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:03:27.145829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59018 (local address 127.0.98.62:40301)
0619 09:03:27.146176 (+ 347us) server_negotiation.cc:206] Beginning negotiation
0619 09:03:27.146186 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 09:03:27.146276 (+ 90us) server_negotiation.cc:407] Connection header received
0619 09:03:27.146401 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:03:27.146408 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:03:27.146462 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:03:27.146593 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:03:27.147322 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:03:27.148666 (+ 1344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:03:27.149813 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:03:27.150411 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:03:27.150804 (+ 393us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:03:27.153157 (+ 2353us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:03:27.153161 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:03:27.153162 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:03:27.153173 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:03:27.155123 (+ 1950us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:03:27.155470 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:03:27.155473 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:03:27.155473 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:03:27.155587 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:03:27.155755 (+ 168us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:03:27.155756 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:03:27.155756 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:03:27.156062 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:03:27.156225 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0619 09:03:27.156296 (+ 71us) server_negotiation.cc:299] Negotiation successful
0619 09:03:27.156429 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":292,"thread_start_us":70,"threads_started":1}
I20260619 09:03:27.157670 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 09:03:27.163439 14095 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 09:03:27.163915 14095 catalog_manager.cc:1504] Loading token signing keys...
I20260619 09:03:27.176957 14095 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 28f6b889c2564926b5cb219e48192ef2: Generated new TSK 0
I20260619 09:03:27.177119 14095 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 09:03:27.211542 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ez5NB/build/release/bin/kudu with pid 14018
2026-06-19T09:03:27Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1243 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859658134510-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 09:03:27 dist-test-slave-mtdr krb5kdc[14113](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 09:03:27 dist-test-slave-mtdr krb5kdc[14113](info): set up 2 sockets
Jun 19 09:03:27 dist-test-slave-mtdr krb5kdc[14113](info): commencing operation
krb5kdc: starting...
W20260619 09:03:28.166604 392 mini_kdc.cc:121] Time spent starting KDC: real 0.938s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859658134510-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859658134510-392-0/krb5kdc/test-user.keytab.
Jun 19 09:03:28 dist-test-slave-mtdr krb5kdc[14113](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859808, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T09:03:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T09:03:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859658134510-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859658134510-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 09:03:28.276806 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ez5NB/build/release/bin/kudu
/tmp/dist-test-task4ez5NB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859658134510-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859658134510-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859658134510-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859658134510-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42391
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859658134510-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42391
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859658134510-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859658134510-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859658134510-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 09:03:28.353555 14126 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 09:03:28.353989 14126 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 09:03:28.354012 14126 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 09:03:28.355217 14126 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 09:03:28.355257 14126 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 09:03:28.355269 14126 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 09:03:28.355280 14126 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 09:03:28.355291 14126 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 09:03:28.356680 14126 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859658134510-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859658134510-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42391
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859658134510-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42391
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859658134510-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14126
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859658134510-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 09:03:28.356861 14126 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 09:03:28.357026 14126 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 09:03:28.361444 14134 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:03:28.361544 14132 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:03:28.361550 14131 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:03:28.361631 14126 server_base.cc:1034] running on GCE node
I20260619 09:03:28.361930 14126 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 09:03:28.362221 14126 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 09:03:28.363394 14126 hybrid_clock.cc:648] HybridClock initialized: now 1781859808363378 us; error 33 us; skew 500 ppm
Jun 19 09:03:28 dist-test-slave-mtdr krb5kdc[14113](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859808, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 09:03:28.365659 14126 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 09:03:28.366048 14126 webserver.cc:460] Webserver started at http://127.0.98.62:45217/ using document root <none> and password file <none>
I20260619 09:03:28.366210 14126 fs_manager.cc:359] Metadata directory not provided
I20260619 09:03:28.366254 14126 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 09:03:28.366344 14126 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 09:03:28.367317 14126 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859658134510-392-0/minicluster-data/master-0/data/instance:
uuid: "0cc5deb37ed24cfc99e60eabbdb09f83"
format_stamp: "Formatted at 2026-06-19 09:03:28 on dist-test-slave-mtdr"
server_key: "ae8de3141b7b6677e87c85df5878a3e4"
server_key_iv: "81239134c3dbc870e290fc8c668b1dcd"
server_key_version: "encryptionkey@0"
I20260619 09:03:28.367596 14126 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859658134510-392-0/minicluster-data/master-0/wal/instance:
uuid: "0cc5deb37ed24cfc99e60eabbdb09f83"
format_stamp: "Formatted at 2026-06-19 09:03:28 on dist-test-slave-mtdr"
server_key: "ae8de3141b7b6677e87c85df5878a3e4"
server_key_iv: "81239134c3dbc870e290fc8c668b1dcd"
server_key_version: "encryptionkey@0"
I20260619 09:03:28.368754 14126 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 09:03:28.369441 14141 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 09:03:28.369597 14126 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 09:03:28.369674 14126 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859658134510-392-0/minicluster-data/master-0/data,/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859658134510-392-0/minicluster-data/master-0/wal
uuid: "0cc5deb37ed24cfc99e60eabbdb09f83"
format_stamp: "Formatted at 2026-06-19 09:03:28 on dist-test-slave-mtdr"
server_key: "ae8de3141b7b6677e87c85df5878a3e4"
server_key_iv: "81239134c3dbc870e290fc8c668b1dcd"
server_key_version: "encryptionkey@0"
I20260619 09:03:28.369727 14126 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859658134510-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859658134510-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859658134510-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 09:03:28.380303 14126 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 09:03:28.382225 14126 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 09:03:28.382324 14126 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 09:03:28.385888 14126 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42391
I20260619 09:03:28.385931 14193 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42391 every 8 connection(s)
I20260619 09:03:28.386221 14126 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859658134510-392-0/minicluster-data/master-0/data/info.pb
I20260619 09:03:28.387136 14194 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 09:03:28.389200 14194 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0cc5deb37ed24cfc99e60eabbdb09f83: Bootstrap starting.
I20260619 09:03:28.389715 14194 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0cc5deb37ed24cfc99e60eabbdb09f83: Neither blocks nor log segments found. Creating new log.
I20260619 09:03:28.389957 14194 log.cc:824] T 00000000000000000000000000000000 P 0cc5deb37ed24cfc99e60eabbdb09f83: Log is configured to *not* fsync() on all Append() calls
I20260619 09:03:28.390558 14194 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0cc5deb37ed24cfc99e60eabbdb09f83: No bootstrap required, opened a new log
I20260619 09:03:28.390981 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ez5NB/build/release/bin/kudu as pid 14126
I20260619 09:03:28.391073 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859658134510-392-0/minicluster-data/master-0/wal/instance
I20260619 09:03:28.391243 392 external_mini_cluster.cc:1428] Setting key 84a7c93e31514c5dc256aff5725289ce
I20260619 09:03:28.392376 14194 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0cc5deb37ed24cfc99e60eabbdb09f83 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0cc5deb37ed24cfc99e60eabbdb09f83" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42391 } }
I20260619 09:03:28.392516 14194 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0cc5deb37ed24cfc99e60eabbdb09f83 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 09:03:28.392550 14194 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0cc5deb37ed24cfc99e60eabbdb09f83 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cc5deb37ed24cfc99e60eabbdb09f83, State: Initialized, Role: FOLLOWER
I20260619 09:03:28.392732 14194 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0cc5deb37ed24cfc99e60eabbdb09f83 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0cc5deb37ed24cfc99e60eabbdb09f83" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42391 } }
I20260619 09:03:28.392817 14194 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0cc5deb37ed24cfc99e60eabbdb09f83 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 09:03:28.392853 14194 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0cc5deb37ed24cfc99e60eabbdb09f83 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 09:03:28.392899 14194 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0cc5deb37ed24cfc99e60eabbdb09f83 [term 0 FOLLOWER]: Advancing to term 1
I20260619 09:03:28.393476 14194 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0cc5deb37ed24cfc99e60eabbdb09f83 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0cc5deb37ed24cfc99e60eabbdb09f83" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42391 } }
I20260619 09:03:28.393569 14194 leader_election.cc:302] T 00000000000000000000000000000000 P 0cc5deb37ed24cfc99e60eabbdb09f83 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0cc5deb37ed24cfc99e60eabbdb09f83; no voters:
I20260619 09:03:28.393720 14194 leader_election.cc:288] T 00000000000000000000000000000000 P 0cc5deb37ed24cfc99e60eabbdb09f83 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 09:03:28.393749 14199 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0cc5deb37ed24cfc99e60eabbdb09f83 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 09:03:28.393841 14199 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0cc5deb37ed24cfc99e60eabbdb09f83 [term 1 LEADER]: Becoming Leader. State: Replica: 0cc5deb37ed24cfc99e60eabbdb09f83, State: Running, Role: LEADER
I20260619 09:03:28.393960 14194 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0cc5deb37ed24cfc99e60eabbdb09f83 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 09:03:28.393968 14199 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0cc5deb37ed24cfc99e60eabbdb09f83 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0cc5deb37ed24cfc99e60eabbdb09f83" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42391 } }
I20260619 09:03:28.394305 14199 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0cc5deb37ed24cfc99e60eabbdb09f83 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0cc5deb37ed24cfc99e60eabbdb09f83. Latest consensus state: current_term: 1 leader_uuid: "0cc5deb37ed24cfc99e60eabbdb09f83" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0cc5deb37ed24cfc99e60eabbdb09f83" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42391 } } }
I20260619 09:03:28.394511 14199 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0cc5deb37ed24cfc99e60eabbdb09f83 [sys.catalog]: This master's current role is: LEADER
I20260619 09:03:28.394446 14200 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0cc5deb37ed24cfc99e60eabbdb09f83 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0cc5deb37ed24cfc99e60eabbdb09f83" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0cc5deb37ed24cfc99e60eabbdb09f83" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42391 } } }
I20260619 09:03:28.394903 14200 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0cc5deb37ed24cfc99e60eabbdb09f83 [sys.catalog]: This master's current role is: LEADER
I20260619 09:03:28.395145 14208 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 09:03:28.395637 14208 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 09:03:28.397123 14208 catalog_manager.cc:1347] Generated new cluster ID: 176104d92ec84d048cf04679fb04c712
I20260619 09:03:28.397174 14208 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 19 09:03:28 dist-test-slave-mtdr krb5kdc[14113](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859808, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 09:03:28.401063 14208 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 09:03:28.401204 14198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:03:28.391680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50318 (local address 127.0.98.62:42391)
0619 09:03:28.391863 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0619 09:03:28.391871 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 09:03:28.391894 (+ 23us) server_negotiation.cc:407] Connection header received
0619 09:03:28.392346 (+ 452us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:03:28.392352 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:03:28.392394 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:03:28.392495 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:03:28.393253 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:03:28.394360 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:03:28.395445 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:03:28.395911 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:03:28.396005 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:03:28.398227 (+ 2222us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:03:28.398230 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:03:28.398231 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:03:28.398241 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:03:28.399732 (+ 1491us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:03:28.400032 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:03:28.400035 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:03:28.400035 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:03:28.400141 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:03:28.400318 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:03:28.400318 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:03:28.400319 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:03:28.400537 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:03:28.400670 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0619 09:03:28.400771 (+ 101us) server_negotiation.cc:299] Negotiation successful
0619 09:03:28.400894 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":100,"threads_started":1}
I20260619 09:03:28.401685 14208 catalog_manager.cc:1504] Loading token signing keys...
I20260619 09:03:28.405728 14208 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0cc5deb37ed24cfc99e60eabbdb09f83: Generated new TSK 0
I20260619 09:03:28.405879 14208 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859658134510-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859658134510-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 09:03:28.434182 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ez5NB/build/release/bin/kudu
/tmp/dist-test-task4ez5NB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859658134510-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859658134510-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859658134510-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859658134510-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42391
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859658134510-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859658134510-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859658134510-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859658134510-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 09:03:28.512039 14220 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 09:03:28.512236 14220 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 09:03:28.512259 14220 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 09:03:28.513350 14220 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 09:03:28.513388 14220 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 09:03:28.513412 14220 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 09:03:28.514768 14220 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859658134510-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859658134510-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859658134510-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859658134510-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42391
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14220
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859658134510-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 09:03:28.514988 14220 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 09:03:28.515183 14220 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 09:03:28.519898 14225 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:03:28.519953 14228 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:03:28.519953 14226 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:03:28.520169 14220 server_base.cc:1034] running on GCE node
I20260619 09:03:28.520382 14220 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 09:03:28.520582 14220 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 09:03:28.521723 14220 hybrid_clock.cc:648] HybridClock initialized: now 1781859808521725 us; error 42 us; skew 500 ppm
Jun 19 09:03:28 dist-test-slave-mtdr krb5kdc[14113](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859808, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 09:03:28.524020 14220 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 09:03:28.524422 14220 webserver.cc:460] Webserver started at http://127.0.98.1:41443/ using document root <none> and password file <none>
I20260619 09:03:28.524602 14220 fs_manager.cc:359] Metadata directory not provided
I20260619 09:03:28.524647 14220 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 09:03:28.524765 14220 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 09:03:28.525718 14220 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859658134510-392-0/minicluster-data/ts-0/data/instance:
uuid: "2f05f67f28604e8faba8ecc8e8eb9569"
format_stamp: "Formatted at 2026-06-19 09:03:28 on dist-test-slave-mtdr"
server_key: "0941bc27c222aac830e412ad20e2321b"
server_key_iv: "1bd5470b474e5ba2fce4f44a95a75da8"
server_key_version: "encryptionkey@0"
I20260619 09:03:28.526021 14220 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859658134510-392-0/minicluster-data/ts-0/wal/instance:
uuid: "2f05f67f28604e8faba8ecc8e8eb9569"
format_stamp: "Formatted at 2026-06-19 09:03:28 on dist-test-slave-mtdr"
server_key: "0941bc27c222aac830e412ad20e2321b"
server_key_iv: "1bd5470b474e5ba2fce4f44a95a75da8"
server_key_version: "encryptionkey@0"
I20260619 09:03:28.527230 14220 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 09:03:28.527870 14235 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 09:03:28.528018 14220 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 09:03:28.528093 14220 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859658134510-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859658134510-392-0/minicluster-data/ts-0/wal
uuid: "2f05f67f28604e8faba8ecc8e8eb9569"
format_stamp: "Formatted at 2026-06-19 09:03:28 on dist-test-slave-mtdr"
server_key: "0941bc27c222aac830e412ad20e2321b"
server_key_iv: "1bd5470b474e5ba2fce4f44a95a75da8"
server_key_version: "encryptionkey@0"
I20260619 09:03:28.528165 14220 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859658134510-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859658134510-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859658134510-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 09:03:28.548089 14220 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 09:03:28.550365 14220 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 09:03:28.550488 14220 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 09:03:28.550684 14220 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 09:03:28.550998 14220 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 09:03:28.551038 14220 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 09:03:28.551060 14220 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 09:03:28.551072 14220 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 09:03:28.556802 14220 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:33591
I20260619 09:03:28.556867 14348 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:33591 every 8 connection(s)
I20260619 09:03:28.557207 14220 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859658134510-392-0/minicluster-data/ts-0/data/info.pb
I20260619 09:03:28.558636 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ez5NB/build/release/bin/kudu as pid 14220
I20260619 09:03:28.558740 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859658134510-392-0/minicluster-data/ts-0/wal/instance
I20260619 09:03:28.558952 392 external_mini_cluster.cc:1428] Setting key 236b960de80880e21ace38870ac81831
Jun 19 09:03:28 dist-test-slave-mtdr krb5kdc[14113](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859808, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260619 09:03:28.567373 14198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:03:28.557942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42325 (local address 127.0.98.62:42391)
0619 09:03:28.558004 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0619 09:03:28.558006 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:03:28.558237 (+ 231us) server_negotiation.cc:407] Connection header received
0619 09:03:28.558784 (+ 547us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:03:28.558785 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:03:28.558802 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:03:28.558875 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:03:28.559920 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:03:28.560420 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:03:28.561867 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:03:28.562084 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:03:28.562148 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:03:28.564498 (+ 2350us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:03:28.564500 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:03:28.564500 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:03:28.564508 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:03:28.565790 (+ 1282us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:03:28.566346 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:03:28.566347 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:03:28.566347 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:03:28.566405 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:03:28.566688 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:03:28.566689 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:03:28.566689 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:03:28.566802 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:03:28.566918 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0619 09:03:28.567189 (+ 271us) server_negotiation.cc:299] Negotiation successful
0619 09:03:28.567278 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260619 09:03:28.567533 14351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:03:28.557931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42391 (local address 127.0.98.1:42325)
0619 09:03:28.558095 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0619 09:03:28.558160 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0619 09:03:28.558680 (+ 520us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:03:28.558950 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:03:28.558954 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:03:28.559581 (+ 627us) client_negotiation.cc:189] Negotiated authn=SASL
0619 09:03:28.559831 (+ 250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:03:28.559838 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:03:28.560499 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:03:28.560500 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:03:28.561766 (+ 1266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:03:28.561770 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:03:28.562160 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:03:28.562161 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:03:28.562301 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:03:28.562791 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 09:03:28.562800 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0619 09:03:28.564410 (+ 1610us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 09:03:28.565975 (+ 1565us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:03:28.565983 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:03:28.565985 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:03:28.566251 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:03:28.566464 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:03:28.566465 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:03:28.566465 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:03:28.566606 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:03:28.566912 (+ 306us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 09:03:28.566919 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 09:03:28.567060 (+ 141us) client_negotiation.cc:769] Sending connection context
0619 09:03:28.567159 (+ 99us) client_negotiation.cc:240] Negotiation successful
0619 09:03:28.567316 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":89,"threads_started":1}
I20260619 09:03:28.568042 14349 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42391
I20260619 09:03:28.568135 14349 heartbeater.cc:464] Registering TS with master...
I20260619 09:03:28.568318 14349 heartbeater.cc:511] Master 127.0.98.62:42391 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260619 09:03:28.568831 14158 ts_manager.cc:194] Registered new tserver with Master: 2f05f67f28604e8faba8ecc8e8eb9569 (127.0.98.1:33591)
I20260619 09:03:28.569625 14158 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:42325
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859658134510-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859658134510-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 09:03:28.582034 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ez5NB/build/release/bin/kudu
/tmp/dist-test-task4ez5NB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859658134510-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859658134510-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859658134510-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859658134510-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42391
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859658134510-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859658134510-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859658134510-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859658134510-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 09:03:28.660162 14354 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 09:03:28.660348 14354 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 09:03:28.660369 14354 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 09:03:28.661427 14354 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 09:03:28.661463 14354 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 09:03:28.661485 14354 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 09:03:28.662886 14354 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859658134510-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859658134510-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859658134510-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859658134510-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42391
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14354
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859658134510-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 09:03:28.663064 14354 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 09:03:28.663219 14354 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 09:03:28.667755 14359 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:03:28.667739 14360 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:03:28.667723 14362 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:03:28.668033 14354 server_base.cc:1034] running on GCE node
I20260619 09:03:28.668211 14354 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 09:03:28.668437 14354 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 09:03:28.669572 14354 hybrid_clock.cc:648] HybridClock initialized: now 1781859808669558 us; error 31 us; skew 500 ppm
Jun 19 09:03:28 dist-test-slave-mtdr krb5kdc[14113](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859808, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 09:03:28.671813 14354 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 09:03:28.672211 14354 webserver.cc:460] Webserver started at http://127.0.98.2:35535/ using document root <none> and password file <none>
I20260619 09:03:28.672417 14354 fs_manager.cc:359] Metadata directory not provided
I20260619 09:03:28.672473 14354 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 09:03:28.672587 14354 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 09:03:28.673638 14354 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859658134510-392-0/minicluster-data/ts-1/data/instance:
uuid: "a4857fc38e35489482ef9d9c80a69f4e"
format_stamp: "Formatted at 2026-06-19 09:03:28 on dist-test-slave-mtdr"
server_key: "2493ea30abd67e288b46d0b06900d4be"
server_key_iv: "c8b22e32b4f351ed2951160a56502e03"
server_key_version: "encryptionkey@0"
I20260619 09:03:28.673945 14354 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859658134510-392-0/minicluster-data/ts-1/wal/instance:
uuid: "a4857fc38e35489482ef9d9c80a69f4e"
format_stamp: "Formatted at 2026-06-19 09:03:28 on dist-test-slave-mtdr"
server_key: "2493ea30abd67e288b46d0b06900d4be"
server_key_iv: "c8b22e32b4f351ed2951160a56502e03"
server_key_version: "encryptionkey@0"
I20260619 09:03:28.675134 14354 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 09:03:28.675844 14369 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 09:03:28.676023 14354 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 09:03:28.676102 14354 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859658134510-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859658134510-392-0/minicluster-data/ts-1/wal
uuid: "a4857fc38e35489482ef9d9c80a69f4e"
format_stamp: "Formatted at 2026-06-19 09:03:28 on dist-test-slave-mtdr"
server_key: "2493ea30abd67e288b46d0b06900d4be"
server_key_iv: "c8b22e32b4f351ed2951160a56502e03"
server_key_version: "encryptionkey@0"
I20260619 09:03:28.676153 14354 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859658134510-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859658134510-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859658134510-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 09:03:28.692188 14354 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 09:03:28.693748 14354 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 09:03:28.693848 14354 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 09:03:28.694021 14354 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 09:03:28.694293 14354 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 09:03:28.694324 14354 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 09:03:28.694341 14354 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 09:03:28.694353 14354 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 09:03:28.699759 14354 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:46005
I20260619 09:03:28.699828 14482 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:46005 every 8 connection(s)
I20260619 09:03:28.700073 14354 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859658134510-392-0/minicluster-data/ts-1/data/info.pb
Jun 19 09:03:28 dist-test-slave-mtdr krb5kdc[14113](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859808, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 09:03:28.706066 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ez5NB/build/release/bin/kudu as pid 14354
I20260619 09:03:28.706171 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859658134510-392-0/minicluster-data/ts-1/wal/instance
I20260619 09:03:28.706357 392 external_mini_cluster.cc:1428] Setting key 0eb9c01a81fc5402a16cfa9a432afe94
I20260619 09:03:28.708514 14198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:03:28.700715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35907 (local address 127.0.98.62:42391)
0619 09:03:28.700765 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0619 09:03:28.700768 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 09:03:28.700951 (+ 183us) server_negotiation.cc:407] Connection header received
0619 09:03:28.701440 (+ 489us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:03:28.701442 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:03:28.701453 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:03:28.701507 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:03:28.702395 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:03:28.702931 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:03:28.703874 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:03:28.704072 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:03:28.704126 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:03:28.705949 (+ 1823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:03:28.705950 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:03:28.705950 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:03:28.705957 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:03:28.707251 (+ 1294us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:03:28.707591 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:03:28.707592 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:03:28.707593 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:03:28.707646 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:03:28.707863 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:03:28.707863 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:03:28.707864 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:03:28.707988 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:03:28.708049 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0619 09:03:28.708380 (+ 331us) server_negotiation.cc:299] Negotiation successful
0619 09:03:28.708452 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260619 09:03:28.708714 14485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:03:28.700693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42391 (local address 127.0.98.2:35907)
0619 09:03:28.700838 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0619 09:03:28.700871 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0619 09:03:28.701351 (+ 480us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:03:28.701587 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:03:28.701591 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:03:28.702117 (+ 526us) client_negotiation.cc:189] Negotiated authn=SASL
0619 09:03:28.702306 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:03:28.702310 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:03:28.703003 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:03:28.703004 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:03:28.703797 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:03:28.703799 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:03:28.704135 (+ 336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:03:28.704136 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:03:28.704267 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:03:28.704641 (+ 374us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 09:03:28.704649 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 09:03:28.705867 (+ 1218us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 09:03:28.707323 (+ 1456us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:03:28.707329 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:03:28.707331 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:03:28.707530 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:03:28.707700 (+ 170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:03:28.707701 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:03:28.707701 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:03:28.707801 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:03:28.708075 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 09:03:28.708081 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 09:03:28.708236 (+ 155us) client_negotiation.cc:769] Sending connection context
0619 09:03:28.708341 (+ 105us) client_negotiation.cc:240] Negotiation successful
0619 09:03:28.708492 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":73,"threads_started":1}
I20260619 09:03:28.709190 14483 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42391
I20260619 09:03:28.709280 14483 heartbeater.cc:464] Registering TS with master...
I20260619 09:03:28.709441 14483 heartbeater.cc:511] Master 127.0.98.62:42391 requested a full tablet report, sending...
I20260619 09:03:28.709769 14158 ts_manager.cc:194] Registered new tserver with Master: a4857fc38e35489482ef9d9c80a69f4e (127.0.98.2:46005)
I20260619 09:03:28.710160 14158 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:35907
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859658134510-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859658134510-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 09:03:28.727524 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ez5NB/build/release/bin/kudu
/tmp/dist-test-task4ez5NB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859658134510-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859658134510-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859658134510-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859658134510-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42391
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859658134510-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859658134510-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859658134510-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859658134510-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 09:03:28.804749 14488 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 09:03:28.804936 14488 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 09:03:28.804965 14488 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 09:03:28.806079 14488 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 09:03:28.806118 14488 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 09:03:28.806151 14488 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 09:03:28.807521 14488 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859658134510-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859658134510-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859658134510-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859658134510-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42391
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14488
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859658134510-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 09:03:28.807737 14488 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 09:03:28.807931 14488 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 09:03:28.812516 14496 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:03:28.812506 14493 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:03:28.812507 14494 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:03:28.812647 14488 server_base.cc:1034] running on GCE node
I20260619 09:03:28.812917 14488 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 09:03:28.813129 14488 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 09:03:28.814263 14488 hybrid_clock.cc:648] HybridClock initialized: now 1781859808814250 us; error 26 us; skew 500 ppm
Jun 19 09:03:28 dist-test-slave-mtdr krb5kdc[14113](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859808, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 09:03:28.816473 14488 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 09:03:28.816835 14488 webserver.cc:460] Webserver started at http://127.0.98.3:45903/ using document root <none> and password file <none>
I20260619 09:03:28.817013 14488 fs_manager.cc:359] Metadata directory not provided
I20260619 09:03:28.817056 14488 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 09:03:28.817173 14488 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 09:03:28.818105 14488 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859658134510-392-0/minicluster-data/ts-2/data/instance:
uuid: "8a40e1510e3a47aab0fc73aa2a498625"
format_stamp: "Formatted at 2026-06-19 09:03:28 on dist-test-slave-mtdr"
server_key: "b48097f65193a9116d9d78149127f2e0"
server_key_iv: "22707887f15ccc2f18c1c79b23928b55"
server_key_version: "encryptionkey@0"
I20260619 09:03:28.818429 14488 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859658134510-392-0/minicluster-data/ts-2/wal/instance:
uuid: "8a40e1510e3a47aab0fc73aa2a498625"
format_stamp: "Formatted at 2026-06-19 09:03:28 on dist-test-slave-mtdr"
server_key: "b48097f65193a9116d9d78149127f2e0"
server_key_iv: "22707887f15ccc2f18c1c79b23928b55"
server_key_version: "encryptionkey@0"
I20260619 09:03:28.819727 14488 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 09:03:28.820415 14503 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 09:03:28.820571 14488 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 09:03:28.820649 14488 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859658134510-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859658134510-392-0/minicluster-data/ts-2/wal
uuid: "8a40e1510e3a47aab0fc73aa2a498625"
format_stamp: "Formatted at 2026-06-19 09:03:28 on dist-test-slave-mtdr"
server_key: "b48097f65193a9116d9d78149127f2e0"
server_key_iv: "22707887f15ccc2f18c1c79b23928b55"
server_key_version: "encryptionkey@0"
I20260619 09:03:28.820708 14488 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859658134510-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859658134510-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859658134510-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 09:03:28.833252 14488 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 09:03:28.834888 14488 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 09:03:28.835016 14488 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 09:03:28.835202 14488 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 09:03:28.835456 14488 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 09:03:28.835487 14488 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 09:03:28.835515 14488 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 09:03:28.835537 14488 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 09:03:28.840591 14488 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35365
I20260619 09:03:28.840642 14616 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35365 every 8 connection(s)
I20260619 09:03:28.840906 14488 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859658134510-392-0/minicluster-data/ts-2/data/info.pb
I20260619 09:03:28.841405 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ez5NB/build/release/bin/kudu as pid 14488
I20260619 09:03:28.841500 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859658134510-392-0/minicluster-data/ts-2/wal/instance
I20260619 09:03:28.841665 392 external_mini_cluster.cc:1428] Setting key 9eaabddc7bb9833b47b7523ebb0dd8ca
Jun 19 09:03:28 dist-test-slave-mtdr krb5kdc[14113](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859808, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 09:03:28.849102 14198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:03:28.841553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60787 (local address 127.0.98.62:42391)
0619 09:03:28.841598 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0619 09:03:28.841601 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 09:03:28.841804 (+ 203us) server_negotiation.cc:407] Connection header received
0619 09:03:28.842348 (+ 544us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:03:28.842349 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:03:28.842362 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:03:28.842423 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:03:28.843333 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:03:28.843830 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:03:28.844750 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:03:28.844932 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:03:28.844984 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:03:28.846767 (+ 1783us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:03:28.846769 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:03:28.846769 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:03:28.846776 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:03:28.847978 (+ 1202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:03:28.848294 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:03:28.848294 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:03:28.848295 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:03:28.848338 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:03:28.848552 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:03:28.848553 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:03:28.848553 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:03:28.848676 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:03:28.848748 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0619 09:03:28.848973 (+ 225us) server_negotiation.cc:299] Negotiation successful
0619 09:03:28.849039 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260619 09:03:28.849232 14619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:03:28.841537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42391 (local address 127.0.98.3:60787)
0619 09:03:28.841676 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0619 09:03:28.841714 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 09:03:28.842256 (+ 542us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:03:28.842481 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:03:28.842485 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:03:28.843070 (+ 585us) client_negotiation.cc:189] Negotiated authn=SASL
0619 09:03:28.843254 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:03:28.843258 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:03:28.843908 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:03:28.843909 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:03:28.844672 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:03:28.844674 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:03:28.844989 (+ 315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:03:28.844989 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:03:28.845078 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:03:28.845468 (+ 390us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 09:03:28.845476 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 09:03:28.846693 (+ 1217us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 09:03:28.848038 (+ 1345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:03:28.848041 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:03:28.848043 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:03:28.848241 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:03:28.848388 (+ 147us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:03:28.848388 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:03:28.848388 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:03:28.848467 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:03:28.848737 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 09:03:28.848740 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 09:03:28.848838 (+ 98us) client_negotiation.cc:769] Sending connection context
0619 09:03:28.848909 (+ 71us) client_negotiation.cc:240] Negotiation successful
0619 09:03:28.849010 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":76,"threads_started":1}
I20260619 09:03:28.849648 14617 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42391
I20260619 09:03:28.849738 14617 heartbeater.cc:464] Registering TS with master...
I20260619 09:03:28.849865 14617 heartbeater.cc:511] Master 127.0.98.62:42391 requested a full tablet report, sending...
I20260619 09:03:28.850155 14158 ts_manager.cc:194] Registered new tserver with Master: 8a40e1510e3a47aab0fc73aa2a498625 (127.0.98.3:35365)
I20260619 09:03:28.850504 14158 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:60787
I20260619 09:03:28.853394 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 09:03:28.860248 14198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:03:28.854702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50322 (local address 127.0.98.62:42391)
0619 09:03:28.854755 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0619 09:03:28.854756 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 09:03:28.854851 (+ 95us) server_negotiation.cc:407] Connection header received
0619 09:03:28.854934 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:03:28.854935 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:03:28.854945 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:03:28.854997 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:03:28.855582 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:03:28.856053 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:03:28.856786 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:03:28.856986 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:03:28.857045 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:03:28.858070 (+ 1025us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:03:28.858071 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:03:28.858072 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:03:28.858079 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:03:28.859151 (+ 1072us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:03:28.859453 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:03:28.859454 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:03:28.859455 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:03:28.859540 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:03:28.859720 (+ 180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:03:28.859721 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:03:28.859721 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:03:28.859855 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:03:28.859946 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0619 09:03:28.860088 (+ 142us) server_negotiation.cc:299] Negotiation successful
0619 09:03:28.860159 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260619 09:03:28.861460 14158 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50322:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 09:03:28.862968 14158 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 09:03:28.869156 14629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:03:28.865042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39098 (local address 127.0.98.3:35365)
0619 09:03:28.865228 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0619 09:03:28.865233 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 09:03:28.865243 (+ 10us) server_negotiation.cc:407] Connection header received
0619 09:03:28.865276 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:03:28.865279 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:03:28.865293 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:03:28.865372 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 09:03:28.865531 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:03:28.866582 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:03:28.868104 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:03:28.868828 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:03:28.868902 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:03:28.868956 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0619 09:03:28.869068 (+ 112us) server_negotiation.cc:299] Negotiation successful
0619 09:03:28.869098 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":101,"threads_started":1}
I20260619 09:03:28.869186 14625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:03:28.864759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35365 (local address 127.0.0.1:39098)
0619 09:03:28.864903 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0619 09:03:28.864940 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0619 09:03:28.864998 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:03:28.865386 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:03:28.865389 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:03:28.865391 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 09:03:28.865450 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:03:28.865452 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:03:28.866696 (+ 1244us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:03:28.866697 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:03:28.868011 (+ 1314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:03:28.868013 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:03:28.868900 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:03:28.868901 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:03:28.868972 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:03:28.868986 (+ 14us) client_negotiation.cc:769] Sending connection context
0619 09:03:28.869054 (+ 68us) client_negotiation.cc:240] Negotiation successful
0619 09:03:28.869099 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"mutex_wait_us":7,"spinlock_wait_cycles":768,"thread_start_us":81,"threads_started":1}
I20260619 09:03:28.869491 14628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:03:28.864931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46005 (local address 127.0.0.1:41198)
0619 09:03:28.865121 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0619 09:03:28.865128 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0619 09:03:28.865163 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:03:28.865455 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:03:28.865455 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:03:28.865457 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 09:03:28.865484 (+ 27us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:03:28.865486 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:03:28.866925 (+ 1439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:03:28.866926 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:03:28.868011 (+ 1085us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:03:28.868013 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:03:28.869267 (+ 1254us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:03:28.869268 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:03:28.869322 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:03:28.869325 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 09:03:28.869392 (+ 67us) client_negotiation.cc:240] Negotiation successful
0619 09:03:28.869427 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176,"mutex_wait_us":5,"thread_start_us":64,"threads_started":1}
I20260619 09:03:28.869701 14630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:03:28.865129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41198 (local address 127.0.98.2:46005)
0619 09:03:28.865255 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0619 09:03:28.865259 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 09:03:28.865269 (+ 10us) server_negotiation.cc:407] Connection header received
0619 09:03:28.865303 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:03:28.865305 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:03:28.865317 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:03:28.865388 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 09:03:28.865828 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:03:28.866857 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:03:28.868111 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:03:28.868828 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:03:28.868902 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:03:28.868957 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0619 09:03:28.869586 (+ 629us) server_negotiation.cc:299] Negotiation successful
0619 09:03:28.869627 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98,"thread_start_us":67,"threads_started":1}
I20260619 09:03:28.869764 14626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:03:28.864759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33591 (local address 127.0.0.1:53944)
0619 09:03:28.864901 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0619 09:03:28.864940 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 09:03:28.864999 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:03:28.865481 (+ 482us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:03:28.865481 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:03:28.865483 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 09:03:28.865527 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:03:28.865529 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:03:28.866715 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:03:28.866716 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:03:28.868714 (+ 1998us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:03:28.868716 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:03:28.869515 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:03:28.869516 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:03:28.869582 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:03:28.869586 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 09:03:28.869653 (+ 67us) client_negotiation.cc:240] Negotiation successful
0619 09:03:28.869693 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"mutex_wait_us":1234,"thread_start_us":50,"threads_started":1}
I20260619 09:03:28.869788 14627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:03:28.864860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53944 (local address 127.0.98.1:33591)
0619 09:03:28.865281 (+ 421us) server_negotiation.cc:206] Beginning negotiation
0619 09:03:28.865284 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 09:03:28.865293 (+ 9us) server_negotiation.cc:407] Connection header received
0619 09:03:28.865327 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:03:28.865328 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:03:28.865341 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:03:28.865403 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 09:03:28.865600 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:03:28.866605 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:03:28.868788 (+ 2183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:03:28.869454 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:03:28.869515 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:03:28.869570 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0619 09:03:28.869668 (+ 98us) server_negotiation.cc:299] Negotiation successful
0619 09:03:28.869715 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":389,"thread_start_us":88,"threads_started":1}
I20260619 09:03:28.870167 14551 tablet_service.cc:1467] Processing CreateTablet for tablet 469ee3e27bbe4e7eb4d61e0d6d5ada38 (DEFAULT_TABLE table=test-table [id=0f59c89a016e4612b088b5e93efd513f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 09:03:28.870328 14283 tablet_service.cc:1467] Processing CreateTablet for tablet 469ee3e27bbe4e7eb4d61e0d6d5ada38 (DEFAULT_TABLE table=test-table [id=0f59c89a016e4612b088b5e93efd513f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 09:03:28.870429 14551 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 469ee3e27bbe4e7eb4d61e0d6d5ada38. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 09:03:28.870553 14283 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 469ee3e27bbe4e7eb4d61e0d6d5ada38. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 09:03:28.871456 14417 tablet_service.cc:1467] Processing CreateTablet for tablet 469ee3e27bbe4e7eb4d61e0d6d5ada38 (DEFAULT_TABLE table=test-table [id=0f59c89a016e4612b088b5e93efd513f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 09:03:28.871758 14417 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 469ee3e27bbe4e7eb4d61e0d6d5ada38. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 09:03:28.873184 14631 tablet_bootstrap.cc:492] T 469ee3e27bbe4e7eb4d61e0d6d5ada38 P 2f05f67f28604e8faba8ecc8e8eb9569: Bootstrap starting.
I20260619 09:03:28.873487 14632 tablet_bootstrap.cc:492] T 469ee3e27bbe4e7eb4d61e0d6d5ada38 P 8a40e1510e3a47aab0fc73aa2a498625: Bootstrap starting.
I20260619 09:03:28.873817 14631 tablet_bootstrap.cc:654] T 469ee3e27bbe4e7eb4d61e0d6d5ada38 P 2f05f67f28604e8faba8ecc8e8eb9569: Neither blocks nor log segments found. Creating new log.
I20260619 09:03:28.874078 14632 tablet_bootstrap.cc:654] T 469ee3e27bbe4e7eb4d61e0d6d5ada38 P 8a40e1510e3a47aab0fc73aa2a498625: Neither blocks nor log segments found. Creating new log.
I20260619 09:03:28.874311 14631 log.cc:824] T 469ee3e27bbe4e7eb4d61e0d6d5ada38 P 2f05f67f28604e8faba8ecc8e8eb9569: Log is configured to *not* fsync() on all Append() calls
I20260619 09:03:28.874576 14632 log.cc:824] T 469ee3e27bbe4e7eb4d61e0d6d5ada38 P 8a40e1510e3a47aab0fc73aa2a498625: Log is configured to *not* fsync() on all Append() calls
I20260619 09:03:28.875088 14631 tablet_bootstrap.cc:492] T 469ee3e27bbe4e7eb4d61e0d6d5ada38 P 2f05f67f28604e8faba8ecc8e8eb9569: No bootstrap required, opened a new log
I20260619 09:03:28.875146 14631 ts_tablet_manager.cc:1397] T 469ee3e27bbe4e7eb4d61e0d6d5ada38 P 2f05f67f28604e8faba8ecc8e8eb9569: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 09:03:28.875212 14632 tablet_bootstrap.cc:492] T 469ee3e27bbe4e7eb4d61e0d6d5ada38 P 8a40e1510e3a47aab0fc73aa2a498625: No bootstrap required, opened a new log
I20260619 09:03:28.875275 14632 ts_tablet_manager.cc:1397] T 469ee3e27bbe4e7eb4d61e0d6d5ada38 P 8a40e1510e3a47aab0fc73aa2a498625: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 09:03:28.876492 14631 raft_consensus.cc:348] T 469ee3e27bbe4e7eb4d61e0d6d5ada38 P 2f05f67f28604e8faba8ecc8e8eb9569 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f05f67f28604e8faba8ecc8e8eb9569" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33591 } } peers { permanent_uuid: "8a40e1510e3a47aab0fc73aa2a498625" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35365 } } peers { permanent_uuid: "a4857fc38e35489482ef9d9c80a69f4e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46005 } }
I20260619 09:03:28.876528 14632 raft_consensus.cc:348] T 469ee3e27bbe4e7eb4d61e0d6d5ada38 P 8a40e1510e3a47aab0fc73aa2a498625 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f05f67f28604e8faba8ecc8e8eb9569" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33591 } } peers { permanent_uuid: "8a40e1510e3a47aab0fc73aa2a498625" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35365 } } peers { permanent_uuid: "a4857fc38e35489482ef9d9c80a69f4e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46005 } }
I20260619 09:03:28.876623 14631 raft_consensus.cc:374] T 469ee3e27bbe4e7eb4d61e0d6d5ada38 P 2f05f67f28604e8faba8ecc8e8eb9569 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 09:03:28.876649 14632 raft_consensus.cc:374] T 469ee3e27bbe4e7eb4d61e0d6d5ada38 P 8a40e1510e3a47aab0fc73aa2a498625 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 09:03:28.876654 14631 raft_consensus.cc:729] T 469ee3e27bbe4e7eb4d61e0d6d5ada38 P 2f05f67f28604e8faba8ecc8e8eb9569 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f05f67f28604e8faba8ecc8e8eb9569, State: Initialized, Role: FOLLOWER
I20260619 09:03:28.876689 14632 raft_consensus.cc:729] T 469ee3e27bbe4e7eb4d61e0d6d5ada38 P 8a40e1510e3a47aab0fc73aa2a498625 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a40e1510e3a47aab0fc73aa2a498625, State: Initialized, Role: FOLLOWER
I20260619 09:03:28.876792 14632 consensus_queue.cc:260] T 469ee3e27bbe4e7eb4d61e0d6d5ada38 P 8a40e1510e3a47aab0fc73aa2a498625 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f05f67f28604e8faba8ecc8e8eb9569" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33591 } } peers { permanent_uuid: "8a40e1510e3a47aab0fc73aa2a498625" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35365 } } peers { permanent_uuid: "a4857fc38e35489482ef9d9c80a69f4e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46005 } }
I20260619 09:03:28.876787 14631 consensus_queue.cc:260] T 469ee3e27bbe4e7eb4d61e0d6d5ada38 P 2f05f67f28604e8faba8ecc8e8eb9569 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f05f67f28604e8faba8ecc8e8eb9569" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33591 } } peers { permanent_uuid: "8a40e1510e3a47aab0fc73aa2a498625" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35365 } } peers { permanent_uuid: "a4857fc38e35489482ef9d9c80a69f4e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46005 } }
I20260619 09:03:28.877025 14632 ts_tablet_manager.cc:1428] T 469ee3e27bbe4e7eb4d61e0d6d5ada38 P 8a40e1510e3a47aab0fc73aa2a498625: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 09:03:28.877031 14617 heartbeater.cc:503] Master 127.0.98.62:42391 was elected leader, sending a full tablet report...
I20260619 09:03:28.877076 14349 heartbeater.cc:503] Master 127.0.98.62:42391 was elected leader, sending a full tablet report...
I20260619 09:03:28.877297 14631 ts_tablet_manager.cc:1428] T 469ee3e27bbe4e7eb4d61e0d6d5ada38 P 2f05f67f28604e8faba8ecc8e8eb9569: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 09:03:28.874742 14634 tablet_bootstrap.cc:492] T 469ee3e27bbe4e7eb4d61e0d6d5ada38 P a4857fc38e35489482ef9d9c80a69f4e: Bootstrap starting.
I20260619 09:03:28.878341 14634 tablet_bootstrap.cc:654] T 469ee3e27bbe4e7eb4d61e0d6d5ada38 P a4857fc38e35489482ef9d9c80a69f4e: Neither blocks nor log segments found. Creating new log.
I20260619 09:03:28.878793 14634 log.cc:824] T 469ee3e27bbe4e7eb4d61e0d6d5ada38 P a4857fc38e35489482ef9d9c80a69f4e: Log is configured to *not* fsync() on all Append() calls
I20260619 09:03:28.879397 14634 tablet_bootstrap.cc:492] T 469ee3e27bbe4e7eb4d61e0d6d5ada38 P a4857fc38e35489482ef9d9c80a69f4e: No bootstrap required, opened a new log
I20260619 09:03:28.879444 14634 ts_tablet_manager.cc:1397] T 469ee3e27bbe4e7eb4d61e0d6d5ada38 P a4857fc38e35489482ef9d9c80a69f4e: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260619 09:03:28.880743 14634 raft_consensus.cc:348] T 469ee3e27bbe4e7eb4d61e0d6d5ada38 P a4857fc38e35489482ef9d9c80a69f4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f05f67f28604e8faba8ecc8e8eb9569" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33591 } } peers { permanent_uuid: "8a40e1510e3a47aab0fc73aa2a498625" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35365 } } peers { permanent_uuid: "a4857fc38e35489482ef9d9c80a69f4e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46005 } }
I20260619 09:03:28.880870 14634 raft_consensus.cc:374] T 469ee3e27bbe4e7eb4d61e0d6d5ada38 P a4857fc38e35489482ef9d9c80a69f4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 09:03:28.880892 14634 raft_consensus.cc:729] T 469ee3e27bbe4e7eb4d61e0d6d5ada38 P a4857fc38e35489482ef9d9c80a69f4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4857fc38e35489482ef9d9c80a69f4e, State: Initialized, Role: FOLLOWER
I20260619 09:03:28.880980 14634 consensus_queue.cc:260] T 469ee3e27bbe4e7eb4d61e0d6d5ada38 P a4857fc38e35489482ef9d9c80a69f4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f05f67f28604e8faba8ecc8e8eb9569" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33591 } } peers { permanent_uuid: "8a40e1510e3a47aab0fc73aa2a498625" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35365 } } peers { permanent_uuid: "a4857fc38e35489482ef9d9c80a69f4e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46005 } }
I20260619 09:03:28.881199 14634 ts_tablet_manager.cc:1428] T 469ee3e27bbe4e7eb4d61e0d6d5ada38 P a4857fc38e35489482ef9d9c80a69f4e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 09:03:28.881235 14483 heartbeater.cc:503] Master 127.0.98.62:42391 was elected leader, sending a full tablet report...
I20260619 09:03:28.915087 14637 raft_consensus.cc:482] T 469ee3e27bbe4e7eb4d61e0d6d5ada38 P 2f05f67f28604e8faba8ecc8e8eb9569 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 09:03:28.915206 14637 raft_consensus.cc:504] T 469ee3e27bbe4e7eb4d61e0d6d5ada38 P 2f05f67f28604e8faba8ecc8e8eb9569 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f05f67f28604e8faba8ecc8e8eb9569" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33591 } } peers { permanent_uuid: "8a40e1510e3a47aab0fc73aa2a498625" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35365 } } peers { permanent_uuid: "a4857fc38e35489482ef9d9c80a69f4e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46005 } }
I20260619 09:03:28.915573 14637 leader_election.cc:288] T 469ee3e27bbe4e7eb4d61e0d6d5ada38 P 2f05f67f28604e8faba8ecc8e8eb9569 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a40e1510e3a47aab0fc73aa2a498625 (127.0.98.3:35365), a4857fc38e35489482ef9d9c80a69f4e (127.0.98.2:46005)
I20260619 09:03:28.918367 14630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:03:28.915930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55811 (local address 127.0.98.2:46005)
0619 09:03:28.915970 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0619 09:03:28.915972 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:03:28.916050 (+ 78us) server_negotiation.cc:407] Connection header received
0619 09:03:28.916077 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:03:28.916077 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:03:28.916089 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:03:28.916150 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 09:03:28.916286 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:03:28.916729 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:03:28.917703 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:03:28.918061 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:03:28.918120 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:03:28.918131 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0619 09:03:28.918268 (+ 137us) server_negotiation.cc:299] Negotiation successful
0619 09:03:28.918304 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260619 09:03:28.918387 14629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:03:28.915792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36293 (local address 127.0.98.3:35365)
0619 09:03:28.915850 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0619 09:03:28.915851 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 09:03:28.915860 (+ 9us) server_negotiation.cc:407] Connection header received
0619 09:03:28.915910 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:03:28.915912 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:03:28.915924 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:03:28.916008 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 09:03:28.916210 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:03:28.916735 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:03:28.917704 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:03:28.918075 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:03:28.918137 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:03:28.918146 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0619 09:03:28.918282 (+ 136us) server_negotiation.cc:299] Negotiation successful
0619 09:03:28.918319 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260619 09:03:28.918397 14351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:03:28.915728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35365 (local address 127.0.98.1:36293)
0619 09:03:28.915783 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0619 09:03:28.915794 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 09:03:28.915847 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:03:28.916059 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:03:28.916060 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:03:28.916062 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 09:03:28.916132 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:03:28.916134 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:03:28.916846 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:03:28.916846 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:03:28.917619 (+ 773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:03:28.917621 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:03:28.918144 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:03:28.918145 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:03:28.918223 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:03:28.918226 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 09:03:28.918270 (+ 44us) client_negotiation.cc:240] Negotiation successful
0619 09:03:28.918313 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35,"mutex_wait_us":22}
I20260619 09:03:28.918419 14640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:03:28.915828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46005 (local address 127.0.98.1:55811)
0619 09:03:28.915970 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0619 09:03:28.915978 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0619 09:03:28.916036 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:03:28.916152 (+ 116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:03:28.916152 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:03:28.916153 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 09:03:28.916224 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:03:28.916225 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:03:28.916802 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:03:28.916802 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:03:28.917621 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:03:28.917623 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:03:28.918134 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:03:28.918135 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:03:28.918207 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:03:28.918213 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 09:03:28.918257 (+ 44us) client_negotiation.cc:240] Negotiation successful
0619 09:03:28.918299 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"mutex_wait_us":42,"thread_start_us":95,"threads_started":1}
I20260619 09:03:28.918722 14571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "469ee3e27bbe4e7eb4d61e0d6d5ada38" candidate_uuid: "2f05f67f28604e8faba8ecc8e8eb9569" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a40e1510e3a47aab0fc73aa2a498625" is_pre_election: true
I20260619 09:03:28.918725 14437 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "469ee3e27bbe4e7eb4d61e0d6d5ada38" candidate_uuid: "2f05f67f28604e8faba8ecc8e8eb9569" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4857fc38e35489482ef9d9c80a69f4e" is_pre_election: true
I20260619 09:03:28.918897 14437 raft_consensus.cc:2454] T 469ee3e27bbe4e7eb4d61e0d6d5ada38 P a4857fc38e35489482ef9d9c80a69f4e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2f05f67f28604e8faba8ecc8e8eb9569 in term 0.
I20260619 09:03:28.918897 14571 raft_consensus.cc:2454] T 469ee3e27bbe4e7eb4d61e0d6d5ada38 P 8a40e1510e3a47aab0fc73aa2a498625 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2f05f67f28604e8faba8ecc8e8eb9569 in term 0.
I20260619 09:03:28.919099 14239 leader_election.cc:302] T 469ee3e27bbe4e7eb4d61e0d6d5ada38 P 2f05f67f28604e8faba8ecc8e8eb9569 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f05f67f28604e8faba8ecc8e8eb9569, 8a40e1510e3a47aab0fc73aa2a498625; no voters:
I20260619 09:03:28.919209 14637 raft_consensus.cc:2781] T 469ee3e27bbe4e7eb4d61e0d6d5ada38 P 2f05f67f28604e8faba8ecc8e8eb9569 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 09:03:28.919260 14637 raft_consensus.cc:482] T 469ee3e27bbe4e7eb4d61e0d6d5ada38 P 2f05f67f28604e8faba8ecc8e8eb9569 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 09:03:28.919287 14637 raft_consensus.cc:3037] T 469ee3e27bbe4e7eb4d61e0d6d5ada38 P 2f05f67f28604e8faba8ecc8e8eb9569 [term 0 FOLLOWER]: Advancing to term 1
I20260619 09:03:28.919927 14637 raft_consensus.cc:504] T 469ee3e27bbe4e7eb4d61e0d6d5ada38 P 2f05f67f28604e8faba8ecc8e8eb9569 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f05f67f28604e8faba8ecc8e8eb9569" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33591 } } peers { permanent_uuid: "8a40e1510e3a47aab0fc73aa2a498625" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35365 } } peers { permanent_uuid: "a4857fc38e35489482ef9d9c80a69f4e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46005 } }
I20260619 09:03:28.920032 14637 leader_election.cc:288] T 469ee3e27bbe4e7eb4d61e0d6d5ada38 P 2f05f67f28604e8faba8ecc8e8eb9569 [CANDIDATE]: Term 1 election: Requested vote from peers 8a40e1510e3a47aab0fc73aa2a498625 (127.0.98.3:35365), a4857fc38e35489482ef9d9c80a69f4e (127.0.98.2:46005)
I20260619 09:03:28.920207 14571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "469ee3e27bbe4e7eb4d61e0d6d5ada38" candidate_uuid: "2f05f67f28604e8faba8ecc8e8eb9569" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a40e1510e3a47aab0fc73aa2a498625"
I20260619 09:03:28.920241 14437 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "469ee3e27bbe4e7eb4d61e0d6d5ada38" candidate_uuid: "2f05f67f28604e8faba8ecc8e8eb9569" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4857fc38e35489482ef9d9c80a69f4e"
I20260619 09:03:28.920281 14571 raft_consensus.cc:3037] T 469ee3e27bbe4e7eb4d61e0d6d5ada38 P 8a40e1510e3a47aab0fc73aa2a498625 [term 0 FOLLOWER]: Advancing to term 1
I20260619 09:03:28.920317 14437 raft_consensus.cc:3037] T 469ee3e27bbe4e7eb4d61e0d6d5ada38 P a4857fc38e35489482ef9d9c80a69f4e [term 0 FOLLOWER]: Advancing to term 1
I20260619 09:03:28.921012 14437 raft_consensus.cc:2454] T 469ee3e27bbe4e7eb4d61e0d6d5ada38 P a4857fc38e35489482ef9d9c80a69f4e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2f05f67f28604e8faba8ecc8e8eb9569 in term 1.
I20260619 09:03:28.921015 14571 raft_consensus.cc:2454] T 469ee3e27bbe4e7eb4d61e0d6d5ada38 P 8a40e1510e3a47aab0fc73aa2a498625 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2f05f67f28604e8faba8ecc8e8eb9569 in term 1.
I20260619 09:03:28.921166 14239 leader_election.cc:302] T 469ee3e27bbe4e7eb4d61e0d6d5ada38 P 2f05f67f28604e8faba8ecc8e8eb9569 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f05f67f28604e8faba8ecc8e8eb9569, 8a40e1510e3a47aab0fc73aa2a498625; no voters:
I20260619 09:03:28.921306 14637 raft_consensus.cc:2781] T 469ee3e27bbe4e7eb4d61e0d6d5ada38 P 2f05f67f28604e8faba8ecc8e8eb9569 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 09:03:28.921453 14637 raft_consensus.cc:686] T 469ee3e27bbe4e7eb4d61e0d6d5ada38 P 2f05f67f28604e8faba8ecc8e8eb9569 [term 1 LEADER]: Becoming Leader. State: Replica: 2f05f67f28604e8faba8ecc8e8eb9569, State: Running, Role: LEADER
I20260619 09:03:28.921561 14637 consensus_queue.cc:237] T 469ee3e27bbe4e7eb4d61e0d6d5ada38 P 2f05f67f28604e8faba8ecc8e8eb9569 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f05f67f28604e8faba8ecc8e8eb9569" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33591 } } peers { permanent_uuid: "8a40e1510e3a47aab0fc73aa2a498625" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35365 } } peers { permanent_uuid: "a4857fc38e35489482ef9d9c80a69f4e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46005 } }
I20260619 09:03:28.922369 14157 catalog_manager.cc:5495] T 469ee3e27bbe4e7eb4d61e0d6d5ada38 P 2f05f67f28604e8faba8ecc8e8eb9569 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2f05f67f28604e8faba8ecc8e8eb9569 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "2f05f67f28604e8faba8ecc8e8eb9569" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f05f67f28604e8faba8ecc8e8eb9569" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33591 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8a40e1510e3a47aab0fc73aa2a498625" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a4857fc38e35489482ef9d9c80a69f4e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46005 } health_report { overall_health: UNKNOWN } } }
I20260619 09:03:28.942080 14627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:03:28.940029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53954 (local address 127.0.98.1:33591)
0619 09:03:28.940080 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0619 09:03:28.940082 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:03:28.940091 (+ 9us) server_negotiation.cc:407] Connection header received
0619 09:03:28.940124 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:03:28.940125 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:03:28.940142 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:03:28.940199 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 09:03:28.940295 (+ 96us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:03:28.940747 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:03:28.941451 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:03:28.941650 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:03:28.941713 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:03:28.941831 (+ 118us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 09:03:28.941927 (+ 96us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 09:03:28.941989 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0619 09:03:28.942003 (+ 14us) server_negotiation.cc:299] Negotiation successful
0619 09:03:28.942020 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260619 09:03:28.943795 14437 raft_consensus.cc:1261] T 469ee3e27bbe4e7eb4d61e0d6d5ada38 P a4857fc38e35489482ef9d9c80a69f4e [term 1 FOLLOWER]: Refusing update from remote peer 2f05f67f28604e8faba8ecc8e8eb9569: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 09:03:28.943804 14571 raft_consensus.cc:1261] T 469ee3e27bbe4e7eb4d61e0d6d5ada38 P 8a40e1510e3a47aab0fc73aa2a498625 [term 1 FOLLOWER]: Refusing update from remote peer 2f05f67f28604e8faba8ecc8e8eb9569: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 09:03:28.944135 14637 consensus_queue.cc:1035] T 469ee3e27bbe4e7eb4d61e0d6d5ada38 P 2f05f67f28604e8faba8ecc8e8eb9569 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a40e1510e3a47aab0fc73aa2a498625" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 09:03:28.944278 14641 consensus_queue.cc:1035] T 469ee3e27bbe4e7eb4d61e0d6d5ada38 P 2f05f67f28604e8faba8ecc8e8eb9569 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4857fc38e35489482ef9d9c80a69f4e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 09:03:28.946625 14646 mvcc.cc:205] Tried to move back new op lower bound from 7298497777431912448 to 7298497777343983616. Current Snapshot: MvccSnapshot[applied={T|T < 7298497777431912448}]
I20260619 09:03:28.946794 14645 mvcc.cc:205] Tried to move back new op lower bound from 7298497777431912448 to 7298497777343983616. Current Snapshot: MvccSnapshot[applied={T|T < 7298497777431912448}]
I20260619 09:03:28.948876 14158 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50322:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 09:03:28.948971 14158 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50322:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 09:03:28.949626 14158 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0cc5deb37ed24cfc99e60eabbdb09f83: Sending DeleteTablet for 3 replicas of tablet 469ee3e27bbe4e7eb4d61e0d6d5ada38
I20260619 09:03:28.949898 14283 tablet_service.cc:1514] Processing DeleteTablet for tablet 469ee3e27bbe4e7eb4d61e0d6d5ada38 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 09:03:28 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53944
I20260619 09:03:28.949940 14551 tablet_service.cc:1514] Processing DeleteTablet for tablet 469ee3e27bbe4e7eb4d61e0d6d5ada38 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 09:03:28 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39098
I20260619 09:03:28.950171 14655 tablet_replica.cc:330] T 469ee3e27bbe4e7eb4d61e0d6d5ada38 P 8a40e1510e3a47aab0fc73aa2a498625: stopping tablet replica
I20260619 09:03:28.950228 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ez5NB/build/release/bin/kudu with pid 14220
I20260619 09:03:28.950285 14655 raft_consensus.cc:2229] T 469ee3e27bbe4e7eb4d61e0d6d5ada38 P 8a40e1510e3a47aab0fc73aa2a498625 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 09:03:28.950335 14417 tablet_service.cc:1514] Processing DeleteTablet for tablet 469ee3e27bbe4e7eb4d61e0d6d5ada38 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 09:03:28 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41198
I20260619 09:03:28.951040 14655 raft_consensus.cc:2258] T 469ee3e27bbe4e7eb4d61e0d6d5ada38 P 8a40e1510e3a47aab0fc73aa2a498625 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 09:03:28.951689 14656 tablet_replica.cc:330] T 469ee3e27bbe4e7eb4d61e0d6d5ada38 P a4857fc38e35489482ef9d9c80a69f4e: stopping tablet replica
I20260619 09:03:28.951781 14656 raft_consensus.cc:2229] T 469ee3e27bbe4e7eb4d61e0d6d5ada38 P a4857fc38e35489482ef9d9c80a69f4e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 09:03:28.951818 14655 ts_tablet_manager.cc:1905] T 469ee3e27bbe4e7eb4d61e0d6d5ada38 P 8a40e1510e3a47aab0fc73aa2a498625: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 09:03:28.951867 14656 raft_consensus.cc:2258] T 469ee3e27bbe4e7eb4d61e0d6d5ada38 P a4857fc38e35489482ef9d9c80a69f4e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 09:03:28.952358 14656 ts_tablet_manager.cc:1905] T 469ee3e27bbe4e7eb4d61e0d6d5ada38 P a4857fc38e35489482ef9d9c80a69f4e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 09:03:28.953495 14655 ts_tablet_manager.cc:1918] T 469ee3e27bbe4e7eb4d61e0d6d5ada38 P 8a40e1510e3a47aab0fc73aa2a498625: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 09:03:28.953564 14655 log.cc:1196] T 469ee3e27bbe4e7eb4d61e0d6d5ada38 P 8a40e1510e3a47aab0fc73aa2a498625: Deleting WAL directory at /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859658134510-392-0/minicluster-data/ts-2/wal/wals/469ee3e27bbe4e7eb4d61e0d6d5ada38
I20260619 09:03:28.953884 14655 ts_tablet_manager.cc:1939] T 469ee3e27bbe4e7eb4d61e0d6d5ada38 P 8a40e1510e3a47aab0fc73aa2a498625: Deleting consensus metadata
I20260619 09:03:28.953907 14656 ts_tablet_manager.cc:1918] T 469ee3e27bbe4e7eb4d61e0d6d5ada38 P a4857fc38e35489482ef9d9c80a69f4e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 09:03:28.953956 14656 log.cc:1196] T 469ee3e27bbe4e7eb4d61e0d6d5ada38 P a4857fc38e35489482ef9d9c80a69f4e: Deleting WAL directory at /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859658134510-392-0/minicluster-data/ts-1/wal/wals/469ee3e27bbe4e7eb4d61e0d6d5ada38
I20260619 09:03:28.954227 14656 ts_tablet_manager.cc:1939] T 469ee3e27bbe4e7eb4d61e0d6d5ada38 P a4857fc38e35489482ef9d9c80a69f4e: Deleting consensus metadata
I20260619 09:03:28.954319 14145 catalog_manager.cc:4841] TS 8a40e1510e3a47aab0fc73aa2a498625 (127.0.98.3:35365): tablet 469ee3e27bbe4e7eb4d61e0d6d5ada38 (table test-table [id=0f59c89a016e4612b088b5e93efd513f]) successfully deleted
I20260619 09:03:28.954586 14145 catalog_manager.cc:4841] TS a4857fc38e35489482ef9d9c80a69f4e (127.0.98.2:46005): tablet 469ee3e27bbe4e7eb4d61e0d6d5ada38 (table test-table [id=0f59c89a016e4612b088b5e93efd513f]) successfully deleted
W20260619 09:03:28.955282 14144 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:33591 (error 108)
I20260619 09:03:28.955475 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ez5NB/build/release/bin/kudu with pid 14354
I20260619 09:03:28.955997 14626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:03:28.955588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33591 (local address 127.0.0.1:53958)
0619 09:03:28.955932 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0619 09:03:28.955967 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":327}
W20260619 09:03:28.956192 14144 catalog_manager.cc:4584] TS 2f05f67f28604e8faba8ecc8e8eb9569 (127.0.98.1:33591): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 469ee3e27bbe4e7eb4d61e0d6d5ada38: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33591: connect: Connection refused (error 111)
I20260619 09:03:28.961807 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ez5NB/build/release/bin/kudu with pid 14488
I20260619 09:03:28.965452 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ez5NB/build/release/bin/kudu with pid 14126
I20260619 09:03:28.969708 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ez5NB/build/release/bin/kudu
/tmp/dist-test-task4ez5NB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859658134510-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859658134510-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859658134510-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859658134510-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42391
--webserver_interface=127.0.98.62
--webserver_port=45217
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859658134510-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42391
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859658134510-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859658134510-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859658134510-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 09:03:29.046581 14658 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 09:03:29.046763 14658 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 09:03:29.046784 14658 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 09:03:29.048043 14658 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 09:03:29.048084 14658 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 09:03:29.048097 14658 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 09:03:29.048110 14658 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 09:03:29.048120 14658 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 09:03:29.049552 14658 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859658134510-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859658134510-392-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42391
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859658134510-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42391
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859658134510-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=45217
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14658
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859658134510-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 09:03:29.049759 14658 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 09:03:29.049965 14658 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 09:03:29.057881 14667 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:03:29.058015 14665 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:03:29.058151 14664 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:03:29.058127 14658 server_base.cc:1034] running on GCE node
I20260619 09:03:29.058425 14658 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 09:03:29.058727 14658 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 09:03:29.059866 14658 hybrid_clock.cc:648] HybridClock initialized: now 1781859809059859 us; error 41 us; skew 500 ppm
Jun 19 09:03:29 dist-test-slave-mtdr krb5kdc[14113](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859809, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 09:03:29.062186 14658 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 09:03:29.062606 14658 webserver.cc:460] Webserver started at http://127.0.98.62:45217/ using document root <none> and password file <none>
I20260619 09:03:29.062855 14658 fs_manager.cc:359] Metadata directory not provided
I20260619 09:03:29.062917 14658 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 09:03:29.064330 14658 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 09:03:29.065074 14674 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 09:03:29.065240 14658 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 09:03:29.065325 14658 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859658134510-392-0/minicluster-data/master-0/data,/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859658134510-392-0/minicluster-data/master-0/wal
uuid: "0cc5deb37ed24cfc99e60eabbdb09f83"
format_stamp: "Formatted at 2026-06-19 09:03:28 on dist-test-slave-mtdr"
server_key: "ae8de3141b7b6677e87c85df5878a3e4"
server_key_iv: "81239134c3dbc870e290fc8c668b1dcd"
server_key_version: "encryptionkey@0"
I20260619 09:03:29.065724 14658 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859658134510-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859658134510-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859658134510-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 09:03:29.100098 14658 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 09:03:29.102262 14658 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 09:03:29.102409 14658 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 09:03:29.106353 14658 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42391
I20260619 09:03:29.106432 14726 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42391 every 8 connection(s)
I20260619 09:03:29.106772 14658 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859658134510-392-0/minicluster-data/master-0/data/info.pb
I20260619 09:03:29.108265 14727 sys_catalog.cc:263] Verifying existing consensus state
I20260619 09:03:29.108834 14727 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0cc5deb37ed24cfc99e60eabbdb09f83: Bootstrap starting.
I20260619 09:03:29.114486 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ez5NB/build/release/bin/kudu as pid 14658
I20260619 09:03:29.114842 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ez5NB/build/release/bin/kudu
/tmp/dist-test-task4ez5NB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859658134510-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859658134510-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859658134510-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859658134510-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:33591
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:42391
--webserver_port=41443
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859658134510-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859658134510-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859658134510-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859658134510-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 09:03:29.116818 14727 log.cc:824] T 00000000000000000000000000000000 P 0cc5deb37ed24cfc99e60eabbdb09f83: Log is configured to *not* fsync() on all Append() calls
I20260619 09:03:29.118656 14727 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0cc5deb37ed24cfc99e60eabbdb09f83: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 09:03:29.118918 14727 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0cc5deb37ed24cfc99e60eabbdb09f83: Bootstrap complete.
I20260619 09:03:29.120965 14727 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0cc5deb37ed24cfc99e60eabbdb09f83 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0cc5deb37ed24cfc99e60eabbdb09f83" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42391 } }
I20260619 09:03:29.121199 14727 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0cc5deb37ed24cfc99e60eabbdb09f83 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cc5deb37ed24cfc99e60eabbdb09f83, State: Initialized, Role: FOLLOWER
I20260619 09:03:29.121335 14727 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0cc5deb37ed24cfc99e60eabbdb09f83 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0cc5deb37ed24cfc99e60eabbdb09f83" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42391 } }
I20260619 09:03:29.121388 14727 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0cc5deb37ed24cfc99e60eabbdb09f83 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 09:03:29.121436 14727 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0cc5deb37ed24cfc99e60eabbdb09f83 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 09:03:29.121481 14727 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0cc5deb37ed24cfc99e60eabbdb09f83 [term 1 FOLLOWER]: Advancing to term 2
I20260619 09:03:29.122404 14727 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0cc5deb37ed24cfc99e60eabbdb09f83 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0cc5deb37ed24cfc99e60eabbdb09f83" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42391 } }
I20260619 09:03:29.122493 14727 leader_election.cc:302] T 00000000000000000000000000000000 P 0cc5deb37ed24cfc99e60eabbdb09f83 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0cc5deb37ed24cfc99e60eabbdb09f83; no voters:
I20260619 09:03:29.122632 14727 leader_election.cc:288] T 00000000000000000000000000000000 P 0cc5deb37ed24cfc99e60eabbdb09f83 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260619 09:03:29.122678 14731 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0cc5deb37ed24cfc99e60eabbdb09f83 [term 2 FOLLOWER]: Leader election won for term 2
I20260619 09:03:29.122844 14731 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0cc5deb37ed24cfc99e60eabbdb09f83 [term 2 LEADER]: Becoming Leader. State: Replica: 0cc5deb37ed24cfc99e60eabbdb09f83, State: Running, Role: LEADER
I20260619 09:03:29.122896 14727 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0cc5deb37ed24cfc99e60eabbdb09f83 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 09:03:29.122967 14731 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0cc5deb37ed24cfc99e60eabbdb09f83 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0cc5deb37ed24cfc99e60eabbdb09f83" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42391 } }
I20260619 09:03:29.123209 14731 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0cc5deb37ed24cfc99e60eabbdb09f83 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0cc5deb37ed24cfc99e60eabbdb09f83. Latest consensus state: current_term: 2 leader_uuid: "0cc5deb37ed24cfc99e60eabbdb09f83" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0cc5deb37ed24cfc99e60eabbdb09f83" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42391 } } }
I20260619 09:03:29.123425 14731 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0cc5deb37ed24cfc99e60eabbdb09f83 [sys.catalog]: This master's current role is: LEADER
I20260619 09:03:29.123359 14732 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0cc5deb37ed24cfc99e60eabbdb09f83 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "0cc5deb37ed24cfc99e60eabbdb09f83" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0cc5deb37ed24cfc99e60eabbdb09f83" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42391 } } }
I20260619 09:03:29.123719 14732 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0cc5deb37ed24cfc99e60eabbdb09f83 [sys.catalog]: This master's current role is: LEADER
I20260619 09:03:29.123831 14740 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 09:03:29.124579 14740 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 09:03:29.124717 14740 catalog_manager.cc:1259] Loaded cluster ID: 176104d92ec84d048cf04679fb04c712
I20260619 09:03:29.124949 14740 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 09:03:29.125190 14740 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260619 09:03:29.130213 14740 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260619 09:03:29.132586 14740 catalog_manager.cc:1504] Loading token signing keys...
I20260619 09:03:29.132783 14740 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 0cc5deb37ed24cfc99e60eabbdb09f83: Loaded TSK: 0
I20260619 09:03:29.132987 14740 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 09:03:29.196101 14729 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 09:03:29.196288 14729 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 09:03:29.196311 14729 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 09:03:29.197443 14729 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 09:03:29.197487 14729 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 09:03:29.197513 14729 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 09:03:29.198942 14729 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859658134510-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859658134510-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859658134510-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:33591
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859658134510-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=41443
--tserver_master_addrs=127.0.98.62:42391
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14729
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859658134510-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 09:03:29.199156 14729 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 09:03:29.199385 14729 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 09:03:29.204121 14757 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:03:29.204140 14755 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:03:29.204139 14754 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:03:29.204411 14729 server_base.cc:1034] running on GCE node
I20260619 09:03:29.204600 14729 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 09:03:29.204807 14729 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 09:03:29.205950 14729 hybrid_clock.cc:648] HybridClock initialized: now 1781859809205909 us; error 61 us; skew 500 ppm
Jun 19 09:03:29 dist-test-slave-mtdr krb5kdc[14113](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859809, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 09:03:29.208220 14729 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 09:03:29.208613 14729 webserver.cc:460] Webserver started at http://127.0.98.1:41443/ using document root <none> and password file <none>
I20260619 09:03:29.208817 14729 fs_manager.cc:359] Metadata directory not provided
I20260619 09:03:29.208869 14729 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 09:03:29.210111 14729 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 09:03:29.210721 14764 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 09:03:29.210877 14729 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 09:03:29.210944 14729 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859658134510-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859658134510-392-0/minicluster-data/ts-0/wal
uuid: "2f05f67f28604e8faba8ecc8e8eb9569"
format_stamp: "Formatted at 2026-06-19 09:03:28 on dist-test-slave-mtdr"
server_key: "0941bc27c222aac830e412ad20e2321b"
server_key_iv: "1bd5470b474e5ba2fce4f44a95a75da8"
server_key_version: "encryptionkey@0"
I20260619 09:03:29.211252 14729 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859658134510-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859658134510-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859658134510-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 09:03:29.221227 14729 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 09:03:29.223300 14729 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 09:03:29.223412 14729 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 09:03:29.223616 14729 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 09:03:29.224063 14771 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260619 09:03:29.225927 14729 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260619 09:03:29.225966 14729 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260619 09:03:29.225988 14729 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260619 09:03:29.226572 14729 ts_tablet_manager.cc:610] Registered 1 tablets
I20260619 09:03:29.226603 14729 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260619 09:03:29.226689 14771 tablet_bootstrap.cc:492] T 469ee3e27bbe4e7eb4d61e0d6d5ada38 P 2f05f67f28604e8faba8ecc8e8eb9569: Bootstrap starting.
I20260619 09:03:29.232831 14729 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:33591
I20260619 09:03:29.233153 14729 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859658134510-392-0/minicluster-data/ts-0/data/info.pb
I20260619 09:03:29.236739 14771 log.cc:824] T 469ee3e27bbe4e7eb4d61e0d6d5ada38 P 2f05f67f28604e8faba8ecc8e8eb9569: Log is configured to *not* fsync() on all Append() calls
I20260619 09:03:29.236994 14878 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:33591 every 8 connection(s)
I20260619 09:03:29.239187 14771 tablet_bootstrap.cc:492] T 469ee3e27bbe4e7eb4d61e0d6d5ada38 P 2f05f67f28604e8faba8ecc8e8eb9569: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 09:03:29.239475 14771 tablet_bootstrap.cc:492] T 469ee3e27bbe4e7eb4d61e0d6d5ada38 P 2f05f67f28604e8faba8ecc8e8eb9569: Bootstrap complete.
I20260619 09:03:29.239640 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ez5NB/build/release/bin/kudu as pid 14729
I20260619 09:03:29.239897 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ez5NB/build/release/bin/kudu
/tmp/dist-test-task4ez5NB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859658134510-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859658134510-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859658134510-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859658134510-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:46005
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:42391
--webserver_port=35535
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859658134510-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859658134510-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859658134510-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859658134510-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 09:03:29.240561 14771 ts_tablet_manager.cc:1397] T 469ee3e27bbe4e7eb4d61e0d6d5ada38 P 2f05f67f28604e8faba8ecc8e8eb9569: Time spent bootstrapping tablet: real 0.014s user 0.009s sys 0.004s
I20260619 09:03:29.242316 14771 raft_consensus.cc:348] T 469ee3e27bbe4e7eb4d61e0d6d5ada38 P 2f05f67f28604e8faba8ecc8e8eb9569 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f05f67f28604e8faba8ecc8e8eb9569" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33591 } } peers { permanent_uuid: "8a40e1510e3a47aab0fc73aa2a498625" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35365 } } peers { permanent_uuid: "a4857fc38e35489482ef9d9c80a69f4e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46005 } }
I20260619 09:03:29.242636 14771 raft_consensus.cc:729] T 469ee3e27bbe4e7eb4d61e0d6d5ada38 P 2f05f67f28604e8faba8ecc8e8eb9569 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f05f67f28604e8faba8ecc8e8eb9569, State: Initialized, Role: FOLLOWER
I20260619 09:03:29.242780 14771 consensus_queue.cc:260] T 469ee3e27bbe4e7eb4d61e0d6d5ada38 P 2f05f67f28604e8faba8ecc8e8eb9569 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f05f67f28604e8faba8ecc8e8eb9569" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33591 } } peers { permanent_uuid: "8a40e1510e3a47aab0fc73aa2a498625" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35365 } } peers { permanent_uuid: "a4857fc38e35489482ef9d9c80a69f4e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46005 } }
I20260619 09:03:29.243036 14771 ts_tablet_manager.cc:1428] T 469ee3e27bbe4e7eb4d61e0d6d5ada38 P 2f05f67f28604e8faba8ecc8e8eb9569: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
Jun 19 09:03:29 dist-test-slave-mtdr krb5kdc[14113](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859809, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 09:03:29.246778 14881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:03:29.233721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42391 (local address 127.0.98.1:48383)
0619 09:03:29.236919 (+ 3198us) negotiation.cc:107] Waiting for socket to connect
0619 09:03:29.236968 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0619 09:03:29.237527 (+ 559us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:03:29.237919 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:03:29.237924 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:03:29.238189 (+ 265us) client_negotiation.cc:189] Negotiated authn=SASL
0619 09:03:29.238358 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:03:29.238362 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:03:29.239563 (+ 1201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:03:29.239564 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:03:29.240722 (+ 1158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:03:29.240725 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:03:29.241652 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:03:29.241653 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:03:29.241781 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:03:29.242341 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 09:03:29.242351 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 09:03:29.243715 (+ 1364us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 09:03:29.245359 (+ 1644us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:03:29.245365 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:03:29.245367 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:03:29.245575 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:03:29.245786 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:03:29.245786 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:03:29.245787 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:03:29.245882 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:03:29.246207 (+ 325us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 09:03:29.246213 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 09:03:29.246338 (+ 125us) client_negotiation.cc:769] Sending connection context
0619 09:03:29.246420 (+ 82us) client_negotiation.cc:240] Negotiation successful
0619 09:03:29.246592 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3153,"thread_start_us":86,"threads_started":1}
I20260619 09:03:29.246778 14882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:03:29.233791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48383 (local address 127.0.98.62:42391)
0619 09:03:29.233981 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0619 09:03:29.233988 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 09:03:29.237102 (+ 3114us) server_negotiation.cc:407] Connection header received
0619 09:03:29.237726 (+ 624us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:03:29.237731 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:03:29.237773 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:03:29.237884 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:03:29.238466 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:03:29.239413 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:03:29.240822 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:03:29.241431 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:03:29.241510 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:03:29.243807 (+ 2297us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:03:29.243811 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:03:29.243811 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:03:29.243823 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:03:29.245285 (+ 1462us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:03:29.245644 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:03:29.245646 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:03:29.245647 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:03:29.245730 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:03:29.245934 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:03:29.245935 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:03:29.245935 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:03:29.246145 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:03:29.246213 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0619 09:03:29.246478 (+ 265us) server_negotiation.cc:299] Negotiation successful
0619 09:03:29.246593 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":76,"threads_started":1}
I20260619 09:03:29.247684 14879 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42391
I20260619 09:03:29.247774 14879 heartbeater.cc:464] Registering TS with master...
I20260619 09:03:29.247938 14879 heartbeater.cc:511] Master 127.0.98.62:42391 requested a full tablet report, sending...
I20260619 09:03:29.248559 14691 ts_manager.cc:194] Registered new tserver with Master: 2f05f67f28604e8faba8ecc8e8eb9569 (127.0.98.1:33591)
I20260619 09:03:29.249516 14691 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:48383
I20260619 09:03:29.250474 14879 heartbeater.cc:503] Master 127.0.98.62:42391 was elected leader, sending a full tablet report...
I20260619 09:03:29.253232 14887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:03:29.249758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53972 (local address 127.0.98.1:33591)
0619 09:03:29.249900 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0619 09:03:29.249904 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 09:03:29.250207 (+ 303us) server_negotiation.cc:407] Connection header received
0619 09:03:29.250239 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:03:29.250241 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:03:29.250349 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:03:29.250413 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 09:03:29.250690 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:03:29.251528 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:03:29.252449 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:03:29.252891 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:03:29.252950 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:03:29.253002 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0619 09:03:29.253127 (+ 125us) server_negotiation.cc:299] Negotiation successful
0619 09:03:29.253163 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":76,"threads_started":1}
I20260619 09:03:29.253232 14886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:03:29.249085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33591 (local address 127.0.0.1:53972)
0619 09:03:29.250014 (+ 929us) negotiation.cc:107] Waiting for socket to connect
0619 09:03:29.250070 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0619 09:03:29.250136 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:03:29.250468 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:03:29.250471 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:03:29.250473 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 09:03:29.250528 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:03:29.250530 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:03:29.251600 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:03:29.251601 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:03:29.252375 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:03:29.252376 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:03:29.252960 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:03:29.252961 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:03:29.253037 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:03:29.253049 (+ 12us) client_negotiation.cc:769] Sending connection context
0619 09:03:29.253110 (+ 61us) client_negotiation.cc:240] Negotiation successful
0619 09:03:29.253163 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":902,"thread_start_us":89,"threads_started":1}
I20260619 09:03:29.253716 14812 tablet_service.cc:1514] Processing DeleteTablet for tablet 469ee3e27bbe4e7eb4d61e0d6d5ada38 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 09:03:28 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53972
I20260619 09:03:29.254199 14888 tablet_replica.cc:330] stopping tablet replica
I20260619 09:03:29.254293 14888 raft_consensus.cc:2229] T 469ee3e27bbe4e7eb4d61e0d6d5ada38 P 2f05f67f28604e8faba8ecc8e8eb9569 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 09:03:29.254359 14888 raft_consensus.cc:2258] T 469ee3e27bbe4e7eb4d61e0d6d5ada38 P 2f05f67f28604e8faba8ecc8e8eb9569 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 09:03:29.254631 14888 ts_tablet_manager.cc:1905] T 469ee3e27bbe4e7eb4d61e0d6d5ada38 P 2f05f67f28604e8faba8ecc8e8eb9569: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 09:03:29.256493 14888 ts_tablet_manager.cc:1918] T 469ee3e27bbe4e7eb4d61e0d6d5ada38 P 2f05f67f28604e8faba8ecc8e8eb9569: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 09:03:29.256558 14888 log.cc:1196] T 469ee3e27bbe4e7eb4d61e0d6d5ada38 P 2f05f67f28604e8faba8ecc8e8eb9569: Deleting WAL directory at /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859658134510-392-0/minicluster-data/ts-0/wal/wals/469ee3e27bbe4e7eb4d61e0d6d5ada38
I20260619 09:03:29.256747 14888 ts_tablet_manager.cc:1939] T 469ee3e27bbe4e7eb4d61e0d6d5ada38 P 2f05f67f28604e8faba8ecc8e8eb9569: Deleting consensus metadata
I20260619 09:03:29.257195 14677 catalog_manager.cc:4841] TS 2f05f67f28604e8faba8ecc8e8eb9569 (127.0.98.1:33591): tablet 469ee3e27bbe4e7eb4d61e0d6d5ada38 (table test-table [id=0f59c89a016e4612b088b5e93efd513f]) successfully deleted
W20260619 09:03:29.322424 14884 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 09:03:29.322603 14884 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 09:03:29.322625 14884 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 09:03:29.323776 14884 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 09:03:29.323822 14884 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 09:03:29.323848 14884 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 09:03:29.325245 14884 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859658134510-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859658134510-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859658134510-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:46005
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859658134510-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=35535
--tserver_master_addrs=127.0.98.62:42391
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14884
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859658134510-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 09:03:29.325423 14884 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 09:03:29.325582 14884 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 09:03:29.330237 14893 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:03:29.330260 14896 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:03:29.330295 14894 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:03:29.330322 14884 server_base.cc:1034] running on GCE node
I20260619 09:03:29.330603 14884 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 09:03:29.330799 14884 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 09:03:29.331960 14884 hybrid_clock.cc:648] HybridClock initialized: now 1781859809331949 us; error 29 us; skew 500 ppm
Jun 19 09:03:29 dist-test-slave-mtdr krb5kdc[14113](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859809, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 09:03:29.334254 14884 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 09:03:29.334637 14884 webserver.cc:460] Webserver started at http://127.0.98.2:35535/ using document root <none> and password file <none>
I20260619 09:03:29.334846 14884 fs_manager.cc:359] Metadata directory not provided
I20260619 09:03:29.334903 14884 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 09:03:29.336099 14884 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 09:03:29.336563 14903 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 09:03:29.336712 14884 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260619 09:03:29.336788 14884 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859658134510-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859658134510-392-0/minicluster-data/ts-1/wal
uuid: "a4857fc38e35489482ef9d9c80a69f4e"
format_stamp: "Formatted at 2026-06-19 09:03:28 on dist-test-slave-mtdr"
server_key: "2493ea30abd67e288b46d0b06900d4be"
server_key_iv: "c8b22e32b4f351ed2951160a56502e03"
server_key_version: "encryptionkey@0"
I20260619 09:03:29.337088 14884 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859658134510-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859658134510-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859658134510-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 09:03:29.345757 14884 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 09:03:29.347368 14884 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 09:03:29.347466 14884 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 09:03:29.347616 14884 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 09:03:29.347868 14884 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 09:03:29.347906 14884 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 09:03:29.347934 14884 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 09:03:29.347951 14884 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 09:03:29.353467 14884 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:46005
I20260619 09:03:29.353574 15016 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:46005 every 8 connection(s)
I20260619 09:03:29.353930 14884 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859658134510-392-0/minicluster-data/ts-1/data/info.pb
I20260619 09:03:29.356019 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ez5NB/build/release/bin/kudu as pid 14884
I20260619 09:03:29.356261 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ez5NB/build/release/bin/kudu
/tmp/dist-test-task4ez5NB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859658134510-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859658134510-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859658134510-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859658134510-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:35365
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:42391
--webserver_port=45903
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859658134510-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859658134510-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859658134510-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859658134510-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 09:03:29 dist-test-slave-mtdr krb5kdc[14113](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859809, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 09:03:29.363168 14882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:03:29.354566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39117 (local address 127.0.98.62:42391)
0619 09:03:29.354615 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0619 09:03:29.354617 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:03:29.354839 (+ 222us) server_negotiation.cc:407] Connection header received
0619 09:03:29.355401 (+ 562us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:03:29.355402 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:03:29.355416 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:03:29.355481 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:03:29.356444 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:03:29.356942 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:03:29.358307 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:03:29.358514 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:03:29.358573 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:03:29.360735 (+ 2162us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:03:29.360737 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:03:29.360737 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:03:29.360744 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:03:29.361944 (+ 1200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:03:29.362291 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:03:29.362292 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:03:29.362292 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:03:29.362340 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:03:29.362544 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:03:29.362545 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:03:29.362545 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:03:29.362647 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:03:29.362700 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0619 09:03:29.362997 (+ 297us) server_negotiation.cc:299] Negotiation successful
0619 09:03:29.363078 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260619 09:03:29.363318 15019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:03:29.354561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42391 (local address 127.0.98.2:39117)
0619 09:03:29.354721 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0619 09:03:29.354755 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0619 09:03:29.355275 (+ 520us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:03:29.355535 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:03:29.355538 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:03:29.356060 (+ 522us) client_negotiation.cc:189] Negotiated authn=SASL
0619 09:03:29.356281 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:03:29.356287 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:03:29.357023 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:03:29.357024 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:03:29.358218 (+ 1194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:03:29.358221 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:03:29.358592 (+ 371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:03:29.358593 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:03:29.358731 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:03:29.359220 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 09:03:29.359229 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0619 09:03:29.360652 (+ 1423us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 09:03:29.362015 (+ 1363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:03:29.362021 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:03:29.362023 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:03:29.362228 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:03:29.362393 (+ 165us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:03:29.362394 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:03:29.362394 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:03:29.362490 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:03:29.362699 (+ 209us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 09:03:29.362705 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 09:03:29.362867 (+ 162us) client_negotiation.cc:769] Sending connection context
0619 09:03:29.362970 (+ 103us) client_negotiation.cc:240] Negotiation successful
0619 09:03:29.363115 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":91,"threads_started":1}
I20260619 09:03:29.363708 15017 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42391
I20260619 09:03:29.363797 15017 heartbeater.cc:464] Registering TS with master...
I20260619 09:03:29.363965 15017 heartbeater.cc:511] Master 127.0.98.62:42391 requested a full tablet report, sending...
I20260619 09:03:29.364295 14691 ts_manager.cc:194] Registered new tserver with Master: a4857fc38e35489482ef9d9c80a69f4e (127.0.98.2:46005)
I20260619 09:03:29.364651 14691 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:39117
W20260619 09:03:29.433612 15020 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 09:03:29.433800 15020 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 09:03:29.433831 15020 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 09:03:29.434933 15020 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 09:03:29.434973 15020 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 09:03:29.435005 15020 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 09:03:29.436360 15020 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859658134510-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859658134510-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859658134510-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:35365
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859658134510-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=45903
--tserver_master_addrs=127.0.98.62:42391
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15020
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859658134510-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 09:03:29.436573 15020 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 09:03:29.436767 15020 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 09:03:29.441331 15028 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:03:29.441458 15025 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:03:29.441350 15026 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:03:29.441510 15020 server_base.cc:1034] running on GCE node
I20260619 09:03:29.441744 15020 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 09:03:29.441948 15020 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 09:03:29.443075 15020 hybrid_clock.cc:648] HybridClock initialized: now 1781859809443067 us; error 27 us; skew 500 ppm
Jun 19 09:03:29 dist-test-slave-mtdr krb5kdc[14113](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859809, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 09:03:29.445277 15020 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 09:03:29.445665 15020 webserver.cc:460] Webserver started at http://127.0.98.3:45903/ using document root <none> and password file <none>
I20260619 09:03:29.445842 15020 fs_manager.cc:359] Metadata directory not provided
I20260619 09:03:29.445896 15020 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 09:03:29.447124 15020 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260619 09:03:29.447638 15035 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 09:03:29.447775 15020 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260619 09:03:29.447844 15020 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859658134510-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859658134510-392-0/minicluster-data/ts-2/wal
uuid: "8a40e1510e3a47aab0fc73aa2a498625"
format_stamp: "Formatted at 2026-06-19 09:03:28 on dist-test-slave-mtdr"
server_key: "b48097f65193a9116d9d78149127f2e0"
server_key_iv: "22707887f15ccc2f18c1c79b23928b55"
server_key_version: "encryptionkey@0"
I20260619 09:03:29.448139 15020 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859658134510-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859658134510-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859658134510-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 09:03:29.470005 15020 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 09:03:29.472004 15020 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 09:03:29.472127 15020 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 09:03:29.472349 15020 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 09:03:29.472646 15020 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 09:03:29.472679 15020 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 09:03:29.472702 15020 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 09:03:29.472723 15020 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 09:03:29.477962 15020 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35365
I20260619 09:03:29.478001 15148 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35365 every 8 connection(s)
I20260619 09:03:29.478291 15020 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859658134510-392-0/minicluster-data/ts-2/data/info.pb
I20260619 09:03:29.481120 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ez5NB/build/release/bin/kudu as pid 15020
Jun 19 09:03:29 dist-test-slave-mtdr krb5kdc[14113](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859809, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 09:03:29.487619 14882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:03:29.478926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40311 (local address 127.0.98.62:42391)
0619 09:03:29.478980 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0619 09:03:29.478986 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 09:03:29.479166 (+ 180us) server_negotiation.cc:407] Connection header received
0619 09:03:29.479641 (+ 475us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:03:29.479642 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:03:29.479654 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:03:29.479703 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:03:29.480583 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:03:29.481076 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:03:29.482237 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:03:29.482429 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:03:29.482477 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:03:29.484556 (+ 2079us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:03:29.484558 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:03:29.484558 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:03:29.484564 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:03:29.486054 (+ 1490us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:03:29.486499 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:03:29.486499 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:03:29.486500 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:03:29.486546 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:03:29.486849 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:03:29.486849 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:03:29.486849 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:03:29.486992 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:03:29.487063 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0619 09:03:29.487432 (+ 369us) server_negotiation.cc:299] Negotiation successful
0619 09:03:29.487520 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260619 09:03:29.487737 15153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:03:29.481463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50324 (local address 127.0.98.62:42391)
0619 09:03:29.481567 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0619 09:03:29.481569 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:03:29.481579 (+ 10us) server_negotiation.cc:407] Connection header received
0619 09:03:29.481651 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:03:29.481652 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:03:29.481662 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:03:29.481724 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:03:29.482198 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:03:29.482855 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:03:29.483606 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:03:29.483821 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:03:29.483872 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:03:29.484959 (+ 1087us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:03:29.484960 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:03:29.484960 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:03:29.484964 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:03:29.486359 (+ 1395us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:03:29.486849 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:03:29.486850 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:03:29.486850 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:03:29.486927 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:03:29.487172 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:03:29.487173 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:03:29.487173 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:03:29.487271 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:03:29.487485 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0619 09:03:29.487518 (+ 33us) server_negotiation.cc:299] Negotiation successful
0619 09:03:29.487599 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":24,"server-negotiator.queue_time_us":84,"thread_start_us":47,"threads_started":1}
I20260619 09:03:29.488230 15151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:03:29.478925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42391 (local address 127.0.98.3:40311)
0619 09:03:29.479054 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0619 09:03:29.479088 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0619 09:03:29.479546 (+ 458us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:03:29.479771 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:03:29.479774 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:03:29.480305 (+ 531us) client_negotiation.cc:189] Negotiated authn=SASL
0619 09:03:29.480488 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:03:29.480494 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:03:29.481153 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:03:29.481154 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:03:29.482153 (+ 999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:03:29.482155 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:03:29.482495 (+ 340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:03:29.482496 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:03:29.482628 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:03:29.483132 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 09:03:29.483140 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 09:03:29.484477 (+ 1337us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 09:03:29.486150 (+ 1673us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:03:29.486156 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:03:29.486158 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:03:29.486418 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:03:29.486605 (+ 187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:03:29.486606 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:03:29.486606 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:03:29.486759 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:03:29.487084 (+ 325us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 09:03:29.487091 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 09:03:29.487262 (+ 171us) client_negotiation.cc:769] Sending connection context
0619 09:03:29.487802 (+ 540us) client_negotiation.cc:240] Negotiation successful
0619 09:03:29.487992 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":63,"threads_started":1}
I20260619 09:03:29.488687 15149 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42391
I20260619 09:03:29.488790 15149 heartbeater.cc:464] Registering TS with master...
I20260619 09:03:29.488976 15149 heartbeater.cc:511] Master 127.0.98.62:42391 requested a full tablet report, sending...
I20260619 09:03:29.489363 14691 ts_manager.cc:194] Registered new tserver with Master: 8a40e1510e3a47aab0fc73aa2a498625 (127.0.98.3:35365)
I20260619 09:03:29.489712 14691 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:40311
I20260619 09:03:29.499737 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 09:03:29.502619 15153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:03:29.500140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50334 (local address 127.0.98.62:42391)
0619 09:03:29.500233 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0619 09:03:29.500235 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:03:29.500300 (+ 65us) server_negotiation.cc:407] Connection header received
0619 09:03:29.500382 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:03:29.500383 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:03:29.500438 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:03:29.500502 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 09:03:29.500624 (+ 122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:03:29.501145 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:03:29.501903 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:03:29.502111 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:03:29.502165 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:03:29.502317 (+ 152us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 09:03:29.502428 (+ 111us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 09:03:29.502498 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0619 09:03:29.502521 (+ 23us) server_negotiation.cc:299] Negotiation successful
0619 09:03:29.502544 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 09:03:29.502938 14691 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:50334:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 09:03:29.504568 14691 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 09:03:29.506712 14812 tablet_service.cc:1467] Processing CreateTablet for tablet 5149ddddda2e48cebf0c32621d371916 (DEFAULT_TABLE table=test-table [id=51e9718c7ae44c7dad25e9b25020f809]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 09:03:29.506953 14812 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5149ddddda2e48cebf0c32621d371916. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 09:03:29.508507 14771 tablet_bootstrap.cc:492] T 5149ddddda2e48cebf0c32621d371916 P 2f05f67f28604e8faba8ecc8e8eb9569: Bootstrap starting.
I20260619 09:03:29.508800 14771 tablet_bootstrap.cc:654] T 5149ddddda2e48cebf0c32621d371916 P 2f05f67f28604e8faba8ecc8e8eb9569: Neither blocks nor log segments found. Creating new log.
I20260619 09:03:29.509315 14771 tablet_bootstrap.cc:492] T 5149ddddda2e48cebf0c32621d371916 P 2f05f67f28604e8faba8ecc8e8eb9569: No bootstrap required, opened a new log
I20260619 09:03:29.509377 14771 ts_tablet_manager.cc:1397] T 5149ddddda2e48cebf0c32621d371916 P 2f05f67f28604e8faba8ecc8e8eb9569: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260619 09:03:29.509487 14771 raft_consensus.cc:348] T 5149ddddda2e48cebf0c32621d371916 P 2f05f67f28604e8faba8ecc8e8eb9569 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4857fc38e35489482ef9d9c80a69f4e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46005 } } peers { permanent_uuid: "8a40e1510e3a47aab0fc73aa2a498625" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35365 } } peers { permanent_uuid: "2f05f67f28604e8faba8ecc8e8eb9569" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33591 } }
I20260619 09:03:29.509549 14771 raft_consensus.cc:374] T 5149ddddda2e48cebf0c32621d371916 P 2f05f67f28604e8faba8ecc8e8eb9569 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 09:03:29.509589 14771 raft_consensus.cc:729] T 5149ddddda2e48cebf0c32621d371916 P 2f05f67f28604e8faba8ecc8e8eb9569 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f05f67f28604e8faba8ecc8e8eb9569, State: Initialized, Role: FOLLOWER
I20260619 09:03:29.509636 14771 consensus_queue.cc:260] T 5149ddddda2e48cebf0c32621d371916 P 2f05f67f28604e8faba8ecc8e8eb9569 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4857fc38e35489482ef9d9c80a69f4e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46005 } } peers { permanent_uuid: "8a40e1510e3a47aab0fc73aa2a498625" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35365 } } peers { permanent_uuid: "2f05f67f28604e8faba8ecc8e8eb9569" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33591 } }
I20260619 09:03:29.509714 14771 ts_tablet_manager.cc:1428] T 5149ddddda2e48cebf0c32621d371916 P 2f05f67f28604e8faba8ecc8e8eb9569: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260619 09:03:29.510459 15155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:03:29.506487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35365 (local address 127.0.0.1:39112)
0619 09:03:29.506698 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0619 09:03:29.506708 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 09:03:29.506762 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:03:29.507205 (+ 443us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:03:29.507206 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:03:29.507208 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 09:03:29.507253 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:03:29.507255 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:03:29.508322 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:03:29.508323 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:03:29.509254 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:03:29.509257 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:03:29.510222 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:03:29.510222 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:03:29.510288 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:03:29.510292 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 09:03:29.510340 (+ 48us) client_negotiation.cc:240] Negotiation successful
0619 09:03:29.510381 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":90,"threads_started":1}
I20260619 09:03:29.510558 15157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:03:29.506712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39112 (local address 127.0.98.3:35365)
0619 09:03:29.507033 (+ 321us) server_negotiation.cc:206] Beginning negotiation
0619 09:03:29.507037 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 09:03:29.507046 (+ 9us) server_negotiation.cc:407] Connection header received
0619 09:03:29.507089 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:03:29.507091 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:03:29.507111 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:03:29.507186 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 09:03:29.507311 (+ 125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:03:29.508202 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:03:29.509348 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:03:29.510070 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:03:29.510127 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:03:29.510190 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0619 09:03:29.510359 (+ 169us) server_negotiation.cc:299] Negotiation successful
0619 09:03:29.510464 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":285,"thread_start_us":123,"threads_started":1}
I20260619 09:03:29.511165 14886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:03:29.506397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46005 (local address 127.0.0.1:41204)
0619 09:03:29.506451 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0619 09:03:29.506464 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 09:03:29.506516 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:03:29.506968 (+ 452us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:03:29.506970 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:03:29.506971 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 09:03:29.507022 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:03:29.507025 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:03:29.508283 (+ 1258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:03:29.508284 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:03:29.509834 (+ 1550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:03:29.509837 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:03:29.510944 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:03:29.510945 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:03:29.511006 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:03:29.511009 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 09:03:29.511059 (+ 50us) client_negotiation.cc:240] Negotiation successful
0619 09:03:29.511099 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33,"mutex_wait_us":558}
I20260619 09:03:29.511199 15156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:03:29.506491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41204 (local address 127.0.98.2:46005)
0619 09:03:29.506775 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0619 09:03:29.506780 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 09:03:29.506794 (+ 14us) server_negotiation.cc:407] Connection header received
0619 09:03:29.506854 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:03:29.506856 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:03:29.506885 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:03:29.506961 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 09:03:29.507130 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:03:29.508182 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:03:29.510004 (+ 1822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:03:29.510857 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:03:29.510930 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:03:29.510974 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0619 09:03:29.511085 (+ 111us) server_negotiation.cc:299] Negotiation successful
0619 09:03:29.511119 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":218,"thread_start_us":96,"threads_started":1}
I20260619 09:03:29.511409 15083 tablet_service.cc:1467] Processing CreateTablet for tablet 5149ddddda2e48cebf0c32621d371916 (DEFAULT_TABLE table=test-table [id=51e9718c7ae44c7dad25e9b25020f809]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 09:03:29.511741 15083 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5149ddddda2e48cebf0c32621d371916. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 09:03:29.511835 14951 tablet_service.cc:1467] Processing CreateTablet for tablet 5149ddddda2e48cebf0c32621d371916 (DEFAULT_TABLE table=test-table [id=51e9718c7ae44c7dad25e9b25020f809]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 09:03:29.512145 14951 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5149ddddda2e48cebf0c32621d371916. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 09:03:29.515251 15159 tablet_bootstrap.cc:492] T 5149ddddda2e48cebf0c32621d371916 P 8a40e1510e3a47aab0fc73aa2a498625: Bootstrap starting.
I20260619 09:03:29.515467 15160 tablet_bootstrap.cc:492] T 5149ddddda2e48cebf0c32621d371916 P a4857fc38e35489482ef9d9c80a69f4e: Bootstrap starting.
I20260619 09:03:29.515928 15159 tablet_bootstrap.cc:654] T 5149ddddda2e48cebf0c32621d371916 P 8a40e1510e3a47aab0fc73aa2a498625: Neither blocks nor log segments found. Creating new log.
I20260619 09:03:29.515990 15160 tablet_bootstrap.cc:654] T 5149ddddda2e48cebf0c32621d371916 P a4857fc38e35489482ef9d9c80a69f4e: Neither blocks nor log segments found. Creating new log.
I20260619 09:03:29.516395 15159 log.cc:824] T 5149ddddda2e48cebf0c32621d371916 P 8a40e1510e3a47aab0fc73aa2a498625: Log is configured to *not* fsync() on all Append() calls
I20260619 09:03:29.516438 15160 log.cc:824] T 5149ddddda2e48cebf0c32621d371916 P a4857fc38e35489482ef9d9c80a69f4e: Log is configured to *not* fsync() on all Append() calls
I20260619 09:03:29.517061 15160 tablet_bootstrap.cc:492] T 5149ddddda2e48cebf0c32621d371916 P a4857fc38e35489482ef9d9c80a69f4e: No bootstrap required, opened a new log
I20260619 09:03:29.517112 15160 ts_tablet_manager.cc:1397] T 5149ddddda2e48cebf0c32621d371916 P a4857fc38e35489482ef9d9c80a69f4e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 09:03:29.517171 15159 tablet_bootstrap.cc:492] T 5149ddddda2e48cebf0c32621d371916 P 8a40e1510e3a47aab0fc73aa2a498625: No bootstrap required, opened a new log
I20260619 09:03:29.517277 15159 ts_tablet_manager.cc:1397] T 5149ddddda2e48cebf0c32621d371916 P 8a40e1510e3a47aab0fc73aa2a498625: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 09:03:29.518452 15160 raft_consensus.cc:348] T 5149ddddda2e48cebf0c32621d371916 P a4857fc38e35489482ef9d9c80a69f4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4857fc38e35489482ef9d9c80a69f4e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46005 } } peers { permanent_uuid: "8a40e1510e3a47aab0fc73aa2a498625" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35365 } } peers { permanent_uuid: "2f05f67f28604e8faba8ecc8e8eb9569" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33591 } }
I20260619 09:03:29.518632 15160 raft_consensus.cc:374] T 5149ddddda2e48cebf0c32621d371916 P a4857fc38e35489482ef9d9c80a69f4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 09:03:29.518713 15160 raft_consensus.cc:729] T 5149ddddda2e48cebf0c32621d371916 P a4857fc38e35489482ef9d9c80a69f4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4857fc38e35489482ef9d9c80a69f4e, State: Initialized, Role: FOLLOWER
I20260619 09:03:29.518862 15160 consensus_queue.cc:260] T 5149ddddda2e48cebf0c32621d371916 P a4857fc38e35489482ef9d9c80a69f4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4857fc38e35489482ef9d9c80a69f4e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46005 } } peers { permanent_uuid: "8a40e1510e3a47aab0fc73aa2a498625" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35365 } } peers { permanent_uuid: "2f05f67f28604e8faba8ecc8e8eb9569" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33591 } }
I20260619 09:03:29.518741 15159 raft_consensus.cc:348] T 5149ddddda2e48cebf0c32621d371916 P 8a40e1510e3a47aab0fc73aa2a498625 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4857fc38e35489482ef9d9c80a69f4e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46005 } } peers { permanent_uuid: "8a40e1510e3a47aab0fc73aa2a498625" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35365 } } peers { permanent_uuid: "2f05f67f28604e8faba8ecc8e8eb9569" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33591 } }
I20260619 09:03:29.518960 15159 raft_consensus.cc:374] T 5149ddddda2e48cebf0c32621d371916 P 8a40e1510e3a47aab0fc73aa2a498625 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 09:03:29.519016 15159 raft_consensus.cc:729] T 5149ddddda2e48cebf0c32621d371916 P 8a40e1510e3a47aab0fc73aa2a498625 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a40e1510e3a47aab0fc73aa2a498625, State: Initialized, Role: FOLLOWER
I20260619 09:03:29.519155 15017 heartbeater.cc:503] Master 127.0.98.62:42391 was elected leader, sending a full tablet report...
I20260619 09:03:29.519233 15160 ts_tablet_manager.cc:1428] T 5149ddddda2e48cebf0c32621d371916 P a4857fc38e35489482ef9d9c80a69f4e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 09:03:29.519284 15159 consensus_queue.cc:260] T 5149ddddda2e48cebf0c32621d371916 P 8a40e1510e3a47aab0fc73aa2a498625 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4857fc38e35489482ef9d9c80a69f4e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46005 } } peers { permanent_uuid: "8a40e1510e3a47aab0fc73aa2a498625" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35365 } } peers { permanent_uuid: "2f05f67f28604e8faba8ecc8e8eb9569" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33591 } }
I20260619 09:03:29.519569 15159 ts_tablet_manager.cc:1428] T 5149ddddda2e48cebf0c32621d371916 P 8a40e1510e3a47aab0fc73aa2a498625: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260619 09:03:29.519672 15149 heartbeater.cc:503] Master 127.0.98.62:42391 was elected leader, sending a full tablet report...
W20260619 09:03:29.604498 15018 tablet.cc:2369] T 5149ddddda2e48cebf0c32621d371916 P a4857fc38e35489482ef9d9c80a69f4e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 09:03:29.624794 15164 raft_consensus.cc:482] T 5149ddddda2e48cebf0c32621d371916 P 8a40e1510e3a47aab0fc73aa2a498625 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 09:03:29.624985 15164 raft_consensus.cc:504] T 5149ddddda2e48cebf0c32621d371916 P 8a40e1510e3a47aab0fc73aa2a498625 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4857fc38e35489482ef9d9c80a69f4e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46005 } } peers { permanent_uuid: "8a40e1510e3a47aab0fc73aa2a498625" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35365 } } peers { permanent_uuid: "2f05f67f28604e8faba8ecc8e8eb9569" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33591 } }
I20260619 09:03:29.625356 15164 leader_election.cc:288] T 5149ddddda2e48cebf0c32621d371916 P 8a40e1510e3a47aab0fc73aa2a498625 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a4857fc38e35489482ef9d9c80a69f4e (127.0.98.2:46005), 2f05f67f28604e8faba8ecc8e8eb9569 (127.0.98.1:33591)
I20260619 09:03:29.628098 14887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:03:29.625642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35297 (local address 127.0.98.1:33591)
0619 09:03:29.625680 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0619 09:03:29.625681 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 09:03:29.625763 (+ 82us) server_negotiation.cc:407] Connection header received
0619 09:03:29.625814 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:03:29.625814 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:03:29.625827 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:03:29.625880 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 09:03:29.626012 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:03:29.626462 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:03:29.627406 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:03:29.627775 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:03:29.627830 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:03:29.627840 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0619 09:03:29.627971 (+ 131us) server_negotiation.cc:299] Negotiation successful
0619 09:03:29.628028 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260619 09:03:29.628105 15151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:03:29.625509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46005 (local address 127.0.98.3:37629)
0619 09:03:29.625564 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0619 09:03:29.625576 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 09:03:29.625621 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:03:29.625782 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:03:29.625782 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:03:29.625784 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 09:03:29.625858 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:03:29.625861 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:03:29.626519 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:03:29.626520 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:03:29.627336 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:03:29.627338 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:03:29.627865 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:03:29.627866 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:03:29.627926 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:03:29.627929 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 09:03:29.627976 (+ 47us) client_negotiation.cc:240] Negotiation successful
0619 09:03:29.628022 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28,"mutex_wait_us":3}
I20260619 09:03:29.628099 15156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:03:29.625588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37629 (local address 127.0.98.2:46005)
0619 09:03:29.625632 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0619 09:03:29.625634 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:03:29.625644 (+ 10us) server_negotiation.cc:407] Connection header received
0619 09:03:29.625675 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:03:29.625676 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:03:29.625689 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:03:29.625773 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 09:03:29.625928 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:03:29.626445 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:03:29.627423 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:03:29.627790 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:03:29.627855 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:03:29.627865 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0619 09:03:29.627994 (+ 129us) server_negotiation.cc:299] Negotiation successful
0619 09:03:29.628031 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260619 09:03:29.628118 15165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:03:29.625548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33591 (local address 127.0.98.3:35297)
0619 09:03:29.625704 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0619 09:03:29.625712 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0619 09:03:29.625765 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:03:29.625880 (+ 115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:03:29.625881 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:03:29.625882 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 09:03:29.625950 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:03:29.625952 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:03:29.626535 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:03:29.626536 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:03:29.627327 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:03:29.627329 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:03:29.627847 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:03:29.627848 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:03:29.627912 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:03:29.627916 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 09:03:29.627956 (+ 40us) client_negotiation.cc:240] Negotiation successful
0619 09:03:29.628001 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"mutex_wait_us":3,"thread_start_us":92,"threads_started":1}
I20260619 09:03:29.628427 14971 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5149ddddda2e48cebf0c32621d371916" candidate_uuid: "8a40e1510e3a47aab0fc73aa2a498625" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4857fc38e35489482ef9d9c80a69f4e" is_pre_election: true
I20260619 09:03:29.628593 14971 raft_consensus.cc:2454] T 5149ddddda2e48cebf0c32621d371916 P a4857fc38e35489482ef9d9c80a69f4e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a40e1510e3a47aab0fc73aa2a498625 in term 0.
I20260619 09:03:29.628566 14833 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5149ddddda2e48cebf0c32621d371916" candidate_uuid: "8a40e1510e3a47aab0fc73aa2a498625" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f05f67f28604e8faba8ecc8e8eb9569" is_pre_election: true
I20260619 09:03:29.628692 14833 raft_consensus.cc:2454] T 5149ddddda2e48cebf0c32621d371916 P 2f05f67f28604e8faba8ecc8e8eb9569 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a40e1510e3a47aab0fc73aa2a498625 in term 0.
I20260619 09:03:29.628849 15039 leader_election.cc:302] T 5149ddddda2e48cebf0c32621d371916 P 8a40e1510e3a47aab0fc73aa2a498625 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8a40e1510e3a47aab0fc73aa2a498625, a4857fc38e35489482ef9d9c80a69f4e; no voters:
I20260619 09:03:29.628993 15164 raft_consensus.cc:2781] T 5149ddddda2e48cebf0c32621d371916 P 8a40e1510e3a47aab0fc73aa2a498625 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 09:03:29.629055 15164 raft_consensus.cc:482] T 5149ddddda2e48cebf0c32621d371916 P 8a40e1510e3a47aab0fc73aa2a498625 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 09:03:29.629079 15164 raft_consensus.cc:3037] T 5149ddddda2e48cebf0c32621d371916 P 8a40e1510e3a47aab0fc73aa2a498625 [term 0 FOLLOWER]: Advancing to term 1
I20260619 09:03:29.629763 15164 raft_consensus.cc:504] T 5149ddddda2e48cebf0c32621d371916 P 8a40e1510e3a47aab0fc73aa2a498625 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4857fc38e35489482ef9d9c80a69f4e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46005 } } peers { permanent_uuid: "8a40e1510e3a47aab0fc73aa2a498625" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35365 } } peers { permanent_uuid: "2f05f67f28604e8faba8ecc8e8eb9569" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33591 } }
I20260619 09:03:29.629874 15164 leader_election.cc:288] T 5149ddddda2e48cebf0c32621d371916 P 8a40e1510e3a47aab0fc73aa2a498625 [CANDIDATE]: Term 1 election: Requested vote from peers a4857fc38e35489482ef9d9c80a69f4e (127.0.98.2:46005), 2f05f67f28604e8faba8ecc8e8eb9569 (127.0.98.1:33591)
I20260619 09:03:29.630028 14833 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5149ddddda2e48cebf0c32621d371916" candidate_uuid: "8a40e1510e3a47aab0fc73aa2a498625" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f05f67f28604e8faba8ecc8e8eb9569"
I20260619 09:03:29.630053 14971 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5149ddddda2e48cebf0c32621d371916" candidate_uuid: "8a40e1510e3a47aab0fc73aa2a498625" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4857fc38e35489482ef9d9c80a69f4e"
I20260619 09:03:29.630110 14833 raft_consensus.cc:3037] T 5149ddddda2e48cebf0c32621d371916 P 2f05f67f28604e8faba8ecc8e8eb9569 [term 0 FOLLOWER]: Advancing to term 1
I20260619 09:03:29.630124 14971 raft_consensus.cc:3037] T 5149ddddda2e48cebf0c32621d371916 P a4857fc38e35489482ef9d9c80a69f4e [term 0 FOLLOWER]: Advancing to term 1
I20260619 09:03:29.630790 14971 raft_consensus.cc:2454] T 5149ddddda2e48cebf0c32621d371916 P a4857fc38e35489482ef9d9c80a69f4e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a40e1510e3a47aab0fc73aa2a498625 in term 1.
I20260619 09:03:29.630964 15039 leader_election.cc:302] T 5149ddddda2e48cebf0c32621d371916 P 8a40e1510e3a47aab0fc73aa2a498625 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8a40e1510e3a47aab0fc73aa2a498625, a4857fc38e35489482ef9d9c80a69f4e; no voters:
I20260619 09:03:29.631044 15164 raft_consensus.cc:2781] T 5149ddddda2e48cebf0c32621d371916 P 8a40e1510e3a47aab0fc73aa2a498625 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 09:03:29.631196 15164 raft_consensus.cc:686] T 5149ddddda2e48cebf0c32621d371916 P 8a40e1510e3a47aab0fc73aa2a498625 [term 1 LEADER]: Becoming Leader. State: Replica: 8a40e1510e3a47aab0fc73aa2a498625, State: Running, Role: LEADER
I20260619 09:03:29.631237 14833 raft_consensus.cc:2454] T 5149ddddda2e48cebf0c32621d371916 P 2f05f67f28604e8faba8ecc8e8eb9569 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a40e1510e3a47aab0fc73aa2a498625 in term 1.
I20260619 09:03:29.631345 15164 consensus_queue.cc:237] T 5149ddddda2e48cebf0c32621d371916 P 8a40e1510e3a47aab0fc73aa2a498625 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4857fc38e35489482ef9d9c80a69f4e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46005 } } peers { permanent_uuid: "8a40e1510e3a47aab0fc73aa2a498625" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35365 } } peers { permanent_uuid: "2f05f67f28604e8faba8ecc8e8eb9569" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33591 } }
I20260619 09:03:29.632187 14690 catalog_manager.cc:5495] T 5149ddddda2e48cebf0c32621d371916 P 8a40e1510e3a47aab0fc73aa2a498625 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8a40e1510e3a47aab0fc73aa2a498625 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "8a40e1510e3a47aab0fc73aa2a498625" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4857fc38e35489482ef9d9c80a69f4e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46005 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a40e1510e3a47aab0fc73aa2a498625" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35365 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2f05f67f28604e8faba8ecc8e8eb9569" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33591 } health_report { overall_health: UNKNOWN } } }
I20260619 09:03:29.655916 15157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:03:29.653726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39122 (local address 127.0.98.3:35365)
0619 09:03:29.653767 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0619 09:03:29.653768 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 09:03:29.653777 (+ 9us) server_negotiation.cc:407] Connection header received
0619 09:03:29.653825 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:03:29.653826 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:03:29.653843 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:03:29.653893 (+ 50us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 09:03:29.653985 (+ 92us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:03:29.654441 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:03:29.655296 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:03:29.655501 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:03:29.655565 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:03:29.655660 (+ 95us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 09:03:29.655755 (+ 95us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 09:03:29.655810 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0619 09:03:29.655823 (+ 13us) server_negotiation.cc:299] Negotiation successful
0619 09:03:29.655840 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260619 09:03:29.657701 14833 raft_consensus.cc:1261] T 5149ddddda2e48cebf0c32621d371916 P 2f05f67f28604e8faba8ecc8e8eb9569 [term 1 FOLLOWER]: Refusing update from remote peer 8a40e1510e3a47aab0fc73aa2a498625: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 09:03:29.657709 14971 raft_consensus.cc:1261] T 5149ddddda2e48cebf0c32621d371916 P a4857fc38e35489482ef9d9c80a69f4e [term 1 FOLLOWER]: Refusing update from remote peer 8a40e1510e3a47aab0fc73aa2a498625: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 09:03:29.658056 15164 consensus_queue.cc:1035] T 5149ddddda2e48cebf0c32621d371916 P 8a40e1510e3a47aab0fc73aa2a498625 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4857fc38e35489482ef9d9c80a69f4e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 09:03:29.658205 15166 consensus_queue.cc:1035] T 5149ddddda2e48cebf0c32621d371916 P 8a40e1510e3a47aab0fc73aa2a498625 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f05f67f28604e8faba8ecc8e8eb9569" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 09:03:29.660367 15170 mvcc.cc:205] Tried to move back new op lower bound from 7298497780355751936 to 7298497780251058176. Current Snapshot: MvccSnapshot[applied={T|T < 7298497780355751936}]
I20260619 09:03:29.662382 14690 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:50334:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 09:03:29.662465 14690 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:50334:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 09:03:29.663170 14690 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0cc5deb37ed24cfc99e60eabbdb09f83: Sending DeleteTablet for 3 replicas of tablet 5149ddddda2e48cebf0c32621d371916
I20260619 09:03:29.663446 14812 tablet_service.cc:1514] Processing DeleteTablet for tablet 5149ddddda2e48cebf0c32621d371916 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 09:03:29 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53972
I20260619 09:03:29.663550 14888 tablet_replica.cc:330] T 5149ddddda2e48cebf0c32621d371916 P 2f05f67f28604e8faba8ecc8e8eb9569: stopping tablet replica
I20260619 09:03:29.663607 14888 raft_consensus.cc:2229] T 5149ddddda2e48cebf0c32621d371916 P 2f05f67f28604e8faba8ecc8e8eb9569 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 09:03:29.663579 15083 tablet_service.cc:1514] Processing DeleteTablet for tablet 5149ddddda2e48cebf0c32621d371916 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 09:03:29 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39112
I20260619 09:03:29.663589 14951 tablet_service.cc:1514] Processing DeleteTablet for tablet 5149ddddda2e48cebf0c32621d371916 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 09:03:29 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41204
I20260619 09:03:29.663654 14888 raft_consensus.cc:2258] T 5149ddddda2e48cebf0c32621d371916 P 2f05f67f28604e8faba8ecc8e8eb9569 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 09:03:29.663784 15179 tablet_replica.cc:330] T 5149ddddda2e48cebf0c32621d371916 P 8a40e1510e3a47aab0fc73aa2a498625: stopping tablet replica
I20260619 09:03:29.663784 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ez5NB/build/release/bin/kudu with pid 14729
I20260619 09:03:29.663908 15179 raft_consensus.cc:2229] T 5149ddddda2e48cebf0c32621d371916 P 8a40e1510e3a47aab0fc73aa2a498625 [term 1 LEADER]: Raft consensus shutting down.
I20260619 09:03:29.664007 15179 raft_consensus.cc:2258] T 5149ddddda2e48cebf0c32621d371916 P 8a40e1510e3a47aab0fc73aa2a498625 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 09:03:29.664999 15180 tablet_replica.cc:330] T 5149ddddda2e48cebf0c32621d371916 P a4857fc38e35489482ef9d9c80a69f4e: stopping tablet replica
I20260619 09:03:29.665145 15180 raft_consensus.cc:2229] T 5149ddddda2e48cebf0c32621d371916 P a4857fc38e35489482ef9d9c80a69f4e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 09:03:29.665208 15179 ts_tablet_manager.cc:1905] T 5149ddddda2e48cebf0c32621d371916 P 8a40e1510e3a47aab0fc73aa2a498625: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 09:03:29.665252 15180 raft_consensus.cc:2258] T 5149ddddda2e48cebf0c32621d371916 P a4857fc38e35489482ef9d9c80a69f4e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 09:03:29.666581 15179 ts_tablet_manager.cc:1918] T 5149ddddda2e48cebf0c32621d371916 P 8a40e1510e3a47aab0fc73aa2a498625: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 09:03:29.666643 15179 log.cc:1196] T 5149ddddda2e48cebf0c32621d371916 P 8a40e1510e3a47aab0fc73aa2a498625: Deleting WAL directory at /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859658134510-392-0/minicluster-data/ts-2/wal/wals/5149ddddda2e48cebf0c32621d371916
I20260619 09:03:29.666936 15179 ts_tablet_manager.cc:1939] T 5149ddddda2e48cebf0c32621d371916 P 8a40e1510e3a47aab0fc73aa2a498625: Deleting consensus metadata
I20260619 09:03:29.667295 14678 catalog_manager.cc:4841] TS 8a40e1510e3a47aab0fc73aa2a498625 (127.0.98.3:35365): tablet 5149ddddda2e48cebf0c32621d371916 (table test-table [id=51e9718c7ae44c7dad25e9b25020f809]) successfully deleted
W20260619 09:03:29.668033 14677 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:33591 (error 108)
I20260619 09:03:29.668687 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ez5NB/build/release/bin/kudu with pid 14884
I20260619 09:03:29.670022 14886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:03:29.669878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33591 (local address 127.0.0.1:53986)
0619 09:03:29.669938 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0619 09:03:29.669988 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260619 09:03:29.670143 14677 catalog_manager.cc:4584] TS 2f05f67f28604e8faba8ecc8e8eb9569 (127.0.98.1:33591): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5149ddddda2e48cebf0c32621d371916: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33591: connect: Connection refused (error 111)
I20260619 09:03:29.673173 14886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:03:29.673070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46005 (local address 127.0.0.1:41216)
0619 09:03:29.673117 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0619 09:03:29.673145 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:46005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260619 09:03:29.673930 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ez5NB/build/release/bin/kudu with pid 15020
I20260619 09:03:29.678511 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ez5NB/build/release/bin/kudu with pid 14658
I20260619 09:03:29.683079 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ez5NB/build/release/bin/kudu
/tmp/dist-test-task4ez5NB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859658134510-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859658134510-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859658134510-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859658134510-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42391
--webserver_interface=127.0.98.62
--webserver_port=45217
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859658134510-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42391
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859658134510-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859658134510-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859658134510-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 09:03:29.761688 15182 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 09:03:29.761885 15182 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 09:03:29.761914 15182 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 09:03:29.763041 15182 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 09:03:29.763078 15182 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 09:03:29.763096 15182 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 09:03:29.763113 15182 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 09:03:29.763127 15182 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 09:03:29.764547 15182 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859658134510-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859658134510-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42391
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859658134510-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42391
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859658134510-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=45217
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15182
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859658134510-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 09:03:29.764794 15182 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 09:03:29.764992 15182 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 09:03:29.769595 15188 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:03:29.769589 15187 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:03:29.769649 15182 server_base.cc:1034] running on GCE node
W20260619 09:03:29.769588 15190 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:03:29.770058 15182 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 09:03:29.770313 15182 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 09:03:29.771464 15182 hybrid_clock.cc:648] HybridClock initialized: now 1781859809771451 us; error 26 us; skew 500 ppm
Jun 19 09:03:29 dist-test-slave-mtdr krb5kdc[14113](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859809, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 09:03:29.773801 15182 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 09:03:29.774178 15182 webserver.cc:460] Webserver started at http://127.0.98.62:45217/ using document root <none> and password file <none>
I20260619 09:03:29.774369 15182 fs_manager.cc:359] Metadata directory not provided
I20260619 09:03:29.774411 15182 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 09:03:29.775655 15182 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 09:03:29.776260 15197 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 09:03:29.776428 15182 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 09:03:29.776508 15182 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859658134510-392-0/minicluster-data/master-0/data,/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859658134510-392-0/minicluster-data/master-0/wal
uuid: "0cc5deb37ed24cfc99e60eabbdb09f83"
format_stamp: "Formatted at 2026-06-19 09:03:28 on dist-test-slave-mtdr"
server_key: "ae8de3141b7b6677e87c85df5878a3e4"
server_key_iv: "81239134c3dbc870e290fc8c668b1dcd"
server_key_version: "encryptionkey@0"
I20260619 09:03:29.776794 15182 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859658134510-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859658134510-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859658134510-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 09:03:29.786921 15182 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 09:03:29.788949 15182 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 09:03:29.789067 15182 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 09:03:29.792702 15182 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42391
I20260619 09:03:29.792742 15249 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42391 every 8 connection(s)
I20260619 09:03:29.793076 15182 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859658134510-392-0/minicluster-data/master-0/data/info.pb
I20260619 09:03:29.794559 15250 sys_catalog.cc:263] Verifying existing consensus state
I20260619 09:03:29.795208 15250 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0cc5deb37ed24cfc99e60eabbdb09f83: Bootstrap starting.
I20260619 09:03:29.797240 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ez5NB/build/release/bin/kudu as pid 15182
I20260619 09:03:29.797567 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ez5NB/build/release/bin/kudu
/tmp/dist-test-task4ez5NB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859658134510-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859658134510-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859658134510-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859658134510-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:33591
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:42391
--webserver_port=41443
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859658134510-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859658134510-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859658134510-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859658134510-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 09:03:29.803572 15250 log.cc:824] T 00000000000000000000000000000000 P 0cc5deb37ed24cfc99e60eabbdb09f83: Log is configured to *not* fsync() on all Append() calls
I20260619 09:03:29.805653 15250 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0cc5deb37ed24cfc99e60eabbdb09f83: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 09:03:29.805905 15250 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0cc5deb37ed24cfc99e60eabbdb09f83: Bootstrap complete.
I20260619 09:03:29.808104 15250 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0cc5deb37ed24cfc99e60eabbdb09f83 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0cc5deb37ed24cfc99e60eabbdb09f83" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42391 } }
I20260619 09:03:29.808328 15250 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0cc5deb37ed24cfc99e60eabbdb09f83 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cc5deb37ed24cfc99e60eabbdb09f83, State: Initialized, Role: FOLLOWER
I20260619 09:03:29.808470 15250 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0cc5deb37ed24cfc99e60eabbdb09f83 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0cc5deb37ed24cfc99e60eabbdb09f83" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42391 } }
I20260619 09:03:29.808542 15250 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0cc5deb37ed24cfc99e60eabbdb09f83 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 09:03:29.808588 15250 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0cc5deb37ed24cfc99e60eabbdb09f83 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 09:03:29.808632 15250 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0cc5deb37ed24cfc99e60eabbdb09f83 [term 2 FOLLOWER]: Advancing to term 3
I20260619 09:03:29.809866 15250 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0cc5deb37ed24cfc99e60eabbdb09f83 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0cc5deb37ed24cfc99e60eabbdb09f83" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42391 } }
I20260619 09:03:29.809971 15250 leader_election.cc:302] T 00000000000000000000000000000000 P 0cc5deb37ed24cfc99e60eabbdb09f83 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0cc5deb37ed24cfc99e60eabbdb09f83; no voters:
I20260619 09:03:29.810160 15250 leader_election.cc:288] T 00000000000000000000000000000000 P 0cc5deb37ed24cfc99e60eabbdb09f83 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260619 09:03:29.810223 15254 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0cc5deb37ed24cfc99e60eabbdb09f83 [term 3 FOLLOWER]: Leader election won for term 3
I20260619 09:03:29.810402 15254 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0cc5deb37ed24cfc99e60eabbdb09f83 [term 3 LEADER]: Becoming Leader. State: Replica: 0cc5deb37ed24cfc99e60eabbdb09f83, State: Running, Role: LEADER
I20260619 09:03:29.810425 15250 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0cc5deb37ed24cfc99e60eabbdb09f83 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 09:03:29.810498 15254 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0cc5deb37ed24cfc99e60eabbdb09f83 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0cc5deb37ed24cfc99e60eabbdb09f83" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42391 } }
I20260619 09:03:29.810763 15254 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0cc5deb37ed24cfc99e60eabbdb09f83 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0cc5deb37ed24cfc99e60eabbdb09f83. Latest consensus state: current_term: 3 leader_uuid: "0cc5deb37ed24cfc99e60eabbdb09f83" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0cc5deb37ed24cfc99e60eabbdb09f83" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42391 } } }
I20260619 09:03:29.810966 15254 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0cc5deb37ed24cfc99e60eabbdb09f83 [sys.catalog]: This master's current role is: LEADER
I20260619 09:03:29.810788 15255 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0cc5deb37ed24cfc99e60eabbdb09f83 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "0cc5deb37ed24cfc99e60eabbdb09f83" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0cc5deb37ed24cfc99e60eabbdb09f83" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42391 } } }
I20260619 09:03:29.811201 15255 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0cc5deb37ed24cfc99e60eabbdb09f83 [sys.catalog]: This master's current role is: LEADER
I20260619 09:03:29.811234 15263 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 09:03:29.815482 15263 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 09:03:29.815644 15263 catalog_manager.cc:1259] Loaded cluster ID: 176104d92ec84d048cf04679fb04c712
I20260619 09:03:29.815743 15263 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260619 09:03:29.816080 15263 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fb2c4fe69ad google::LogMessage::Fail() at ??:0
@ 0x7fb2c4fe988a google::LogMessage::SendToLog() at ??:0
@ 0x7fb2c4fe64de google::LogMessage::Flush() at ??:0
@ 0x7fb2c4fe7a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fb2ccf070f1 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7fb2ccf108b9 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fb2c6eb40d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fb2c6ea9954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fb2c7d0d6db start_thread at ??:0
@ 0x7fb2c3ae461f clone at ??:0
W20260619 09:03:29.885779 15252 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 09:03:29.885965 15252 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 09:03:29.885994 15252 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 09:03:29.887136 15252 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 09:03:29.887174 15252 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 09:03:29.887208 15252 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 09:03:29.888595 15252 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859658134510-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859658134510-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859658134510-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:33591
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859658134510-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=41443
--tserver_master_addrs=127.0.98.62:42391
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15252
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859658134510-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 09:03:29.888809 15252 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 09:03:29.889003 15252 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 09:03:29.893615 15289 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:03:29.893640 15286 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:03:29.893759 15287 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:03:29.893751 15252 server_base.cc:1034] running on GCE node
I20260619 09:03:29.893996 15252 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 09:03:29.894207 15252 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 09:03:29.895344 15252 hybrid_clock.cc:648] HybridClock initialized: now 1781859809895334 us; error 25 us; skew 500 ppm
Jun 19 09:03:29 dist-test-slave-mtdr krb5kdc[14113](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859809, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 09:03:29.897660 15252 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 09:03:29.898056 15252 webserver.cc:460] Webserver started at http://127.0.98.1:41443/ using document root <none> and password file <none>
I20260619 09:03:29.898240 15252 fs_manager.cc:359] Metadata directory not provided
I20260619 09:03:29.898298 15252 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 09:03:29.899655 15252 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 09:03:29.900279 15296 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 09:03:29.900440 15252 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 09:03:29.900528 15252 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859658134510-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859658134510-392-0/minicluster-data/ts-0/wal
uuid: "2f05f67f28604e8faba8ecc8e8eb9569"
format_stamp: "Formatted at 2026-06-19 09:03:28 on dist-test-slave-mtdr"
server_key: "0941bc27c222aac830e412ad20e2321b"
server_key_iv: "1bd5470b474e5ba2fce4f44a95a75da8"
server_key_version: "encryptionkey@0"
I20260619 09:03:29.900861 15252 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859658134510-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859658134510-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859658134510-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 09:03:29.928671 15252 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 09:03:29.930794 15252 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 09:03:29.930953 15252 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 09:03:29.931164 15252 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 09:03:29.931576 15303 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260619 09:03:29.933488 15252 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260619 09:03:29.933544 15252 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260619 09:03:29.933579 15252 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260619 09:03:29.934173 15252 ts_tablet_manager.cc:610] Registered 1 tablets
I20260619 09:03:29.934206 15252 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260619 09:03:29.934285 15303 tablet_bootstrap.cc:492] T 5149ddddda2e48cebf0c32621d371916 P 2f05f67f28604e8faba8ecc8e8eb9569: Bootstrap starting.
I20260619 09:03:29.940539 15252 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:33591
I20260619 09:03:29.940675 15410 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:33591 every 8 connection(s)
I20260619 09:03:29.940968 15252 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859658134510-392-0/minicluster-data/ts-0/data/info.pb
I20260619 09:03:29.942672 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ez5NB/build/release/bin/kudu as pid 15252
I20260619 09:03:29.942927 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ez5NB/build/release/bin/kudu
/tmp/dist-test-task4ez5NB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859658134510-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859658134510-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859658134510-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859658134510-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:46005
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:42391
--webserver_port=35535
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859658134510-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859658134510-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859658134510-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859658134510-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 09:03:29.944231 15303 log.cc:824] T 5149ddddda2e48cebf0c32621d371916 P 2f05f67f28604e8faba8ecc8e8eb9569: Log is configured to *not* fsync() on all Append() calls
I20260619 09:03:29.946704 15303 tablet_bootstrap.cc:492] T 5149ddddda2e48cebf0c32621d371916 P 2f05f67f28604e8faba8ecc8e8eb9569: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 09:03:29.946995 15303 tablet_bootstrap.cc:492] T 5149ddddda2e48cebf0c32621d371916 P 2f05f67f28604e8faba8ecc8e8eb9569: Bootstrap complete.
I20260619 09:03:29.948060 15303 ts_tablet_manager.cc:1397] T 5149ddddda2e48cebf0c32621d371916 P 2f05f67f28604e8faba8ecc8e8eb9569: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.010s
I20260619 09:03:29.949435 15303 raft_consensus.cc:348] T 5149ddddda2e48cebf0c32621d371916 P 2f05f67f28604e8faba8ecc8e8eb9569 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4857fc38e35489482ef9d9c80a69f4e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46005 } } peers { permanent_uuid: "8a40e1510e3a47aab0fc73aa2a498625" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35365 } } peers { permanent_uuid: "2f05f67f28604e8faba8ecc8e8eb9569" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33591 } }
I20260619 09:03:29.949755 15303 raft_consensus.cc:729] T 5149ddddda2e48cebf0c32621d371916 P 2f05f67f28604e8faba8ecc8e8eb9569 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f05f67f28604e8faba8ecc8e8eb9569, State: Initialized, Role: FOLLOWER
I20260619 09:03:29.949915 15303 consensus_queue.cc:260] T 5149ddddda2e48cebf0c32621d371916 P 2f05f67f28604e8faba8ecc8e8eb9569 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4857fc38e35489482ef9d9c80a69f4e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46005 } } peers { permanent_uuid: "8a40e1510e3a47aab0fc73aa2a498625" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35365 } } peers { permanent_uuid: "2f05f67f28604e8faba8ecc8e8eb9569" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33591 } }
I20260619 09:03:29.950176 15303 ts_tablet_manager.cc:1428] T 5149ddddda2e48cebf0c32621d371916 P 2f05f67f28604e8faba8ecc8e8eb9569: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
W20260619 09:03:29.998322 15413 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 09:03:29.941529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42391 (local address 127.0.98.1:47271)
0619 09:03:29.941683 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0619 09:03:29.941723 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 09:03:29.942139 (+ 416us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:03:29.998105 (+ 55966us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42391: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":73,"threads_started":1}
W20260619 09:03:29.998633 15411 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:42391 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:42391: Client connection negotiation failed: client connection to 127.0.98.62:42391: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260619 09:03:29.998991 15413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:03:29.998886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42391 (local address 127.0.98.1:37641)
0619 09:03:29.998936 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0619 09:03:29.998970 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260619 09:03:29.999293 15413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:03:29.999210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42391 (local address 127.0.98.1:50911)
0619 09:03:29.999251 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0619 09:03:29.999273 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260619 09:03:29.999396 15411 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 09:03:30.022593 15414 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 09:03:30.022792 15414 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 09:03:30.022832 15414 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 09:03:30.023928 15414 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 09:03:30.023964 15414 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 09:03:30.023989 15414 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 09:03:30.025321 15414 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859658134510-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859658134510-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859658134510-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:46005
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859658134510-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=35535
--tserver_master_addrs=127.0.98.62:42391
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15414
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859658134510-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 09:03:30.025494 15414 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 09:03:30.025668 15414 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 09:03:30.030264 15424 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:03:30.030269 15421 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:03:30.030328 15414 server_base.cc:1034] running on GCE node
W20260619 09:03:30.030298 15422 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:03:30.030611 15414 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 09:03:30.030848 15414 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 09:03:30.032018 15414 hybrid_clock.cc:648] HybridClock initialized: now 1781859810032007 us; error 26 us; skew 500 ppm
Jun 19 09:03:30 dist-test-slave-mtdr krb5kdc[14113](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859810, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 09:03:30.034449 15414 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 09:03:30.034890 15414 webserver.cc:460] Webserver started at http://127.0.98.2:35535/ using document root <none> and password file <none>
I20260619 09:03:30.035099 15414 fs_manager.cc:359] Metadata directory not provided
I20260619 09:03:30.035148 15414 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 09:03:30.036566 15414 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260619 09:03:30.037138 15431 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 09:03:30.037295 15414 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 09:03:30.037375 15414 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859658134510-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859658134510-392-0/minicluster-data/ts-1/wal
uuid: "a4857fc38e35489482ef9d9c80a69f4e"
format_stamp: "Formatted at 2026-06-19 09:03:28 on dist-test-slave-mtdr"
server_key: "2493ea30abd67e288b46d0b06900d4be"
server_key_iv: "c8b22e32b4f351ed2951160a56502e03"
server_key_version: "encryptionkey@0"
I20260619 09:03:30.037739 15414 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859658134510-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859658134510-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859658134510-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 09:03:30.070509 15414 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 09:03:30.072613 15414 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 09:03:30.072753 15414 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 09:03:30.072974 15414 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 09:03:30.073398 15438 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260619 09:03:30.074836 15414 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260619 09:03:30.074889 15414 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260619 09:03:30.074926 15414 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260619 09:03:30.075556 15414 ts_tablet_manager.cc:610] Registered 1 tablets
I20260619 09:03:30.075587 15414 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260619 09:03:30.075724 15438 tablet_bootstrap.cc:492] T 5149ddddda2e48cebf0c32621d371916 P a4857fc38e35489482ef9d9c80a69f4e: Bootstrap starting.
I20260619 09:03:30.077342 15438 log.cc:824] T 5149ddddda2e48cebf0c32621d371916 P a4857fc38e35489482ef9d9c80a69f4e: Log is configured to *not* fsync() on all Append() calls
I20260619 09:03:30.080020 15438 tablet_bootstrap.cc:492] T 5149ddddda2e48cebf0c32621d371916 P a4857fc38e35489482ef9d9c80a69f4e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 09:03:30.080272 15438 tablet_bootstrap.cc:492] T 5149ddddda2e48cebf0c32621d371916 P a4857fc38e35489482ef9d9c80a69f4e: Bootstrap complete.
I20260619 09:03:30.080407 15438 ts_tablet_manager.cc:1397] T 5149ddddda2e48cebf0c32621d371916 P a4857fc38e35489482ef9d9c80a69f4e: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260619 09:03:30.081804 15414 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:46005
I20260619 09:03:30.081854 15546 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:46005 every 8 connection(s)
I20260619 09:03:30.082125 15414 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859658134510-392-0/minicluster-data/ts-1/data/info.pb
I20260619 09:03:30.082019 15438 raft_consensus.cc:348] T 5149ddddda2e48cebf0c32621d371916 P a4857fc38e35489482ef9d9c80a69f4e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4857fc38e35489482ef9d9c80a69f4e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46005 } } peers { permanent_uuid: "8a40e1510e3a47aab0fc73aa2a498625" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35365 } } peers { permanent_uuid: "2f05f67f28604e8faba8ecc8e8eb9569" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33591 } }
I20260619 09:03:30.082304 15438 raft_consensus.cc:729] T 5149ddddda2e48cebf0c32621d371916 P a4857fc38e35489482ef9d9c80a69f4e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4857fc38e35489482ef9d9c80a69f4e, State: Initialized, Role: FOLLOWER
I20260619 09:03:30.082605 15438 consensus_queue.cc:260] T 5149ddddda2e48cebf0c32621d371916 P a4857fc38e35489482ef9d9c80a69f4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4857fc38e35489482ef9d9c80a69f4e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46005 } } peers { permanent_uuid: "8a40e1510e3a47aab0fc73aa2a498625" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35365 } } peers { permanent_uuid: "2f05f67f28604e8faba8ecc8e8eb9569" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33591 } }
I20260619 09:03:30.082886 15438 ts_tablet_manager.cc:1428] T 5149ddddda2e48cebf0c32621d371916 P a4857fc38e35489482ef9d9c80a69f4e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 09:03:30.083091 15550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:03:30.082752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42391 (local address 127.0.98.2:44479)
0619 09:03:30.082904 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0619 09:03:30.082984 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":90,"threads_started":1}
W20260619 09:03:30.083304 15547 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:42391 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:42391: Client connection negotiation failed: client connection to 127.0.98.62:42391: connect: Connection refused (error 111)
I20260619 09:03:30.083539 15550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:03:30.083464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42391 (local address 127.0.98.2:49057)
0619 09:03:30.083493 (+ 29us) negotiation.cc:107] Waiting for socket to connect
0619 09:03:30.083520 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":14}
I20260619 09:03:30.083756 15550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:03:30.083696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42391 (local address 127.0.98.2:58223)
0619 09:03:30.083727 (+ 31us) negotiation.cc:107] Waiting for socket to connect
0619 09:03:30.083745 (+ 18us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
W20260619 09:03:30.083830 15547 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260619 09:03:30.088317 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ez5NB/build/release/bin/kudu as pid 15414
I20260619 09:03:30.088546 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ez5NB/build/release/bin/kudu
/tmp/dist-test-task4ez5NB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859658134510-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859658134510-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859658134510-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859658134510-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:35365
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:42391
--webserver_port=45903
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859658134510-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859658134510-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859658134510-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859658134510-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 09:03:30.165455 15551 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 09:03:30.165654 15551 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 09:03:30.165686 15551 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 09:03:30.166796 15551 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 09:03:30.166858 15551 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 09:03:30.166896 15551 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 09:03:30.168306 15551 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859658134510-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859658134510-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859658134510-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:35365
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859658134510-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=45903
--tserver_master_addrs=127.0.98.62:42391
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15551
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859658134510-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 09:03:30.168525 15551 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 09:03:30.168728 15551 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 09:03:30.173460 15557 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:03:30.173486 15556 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:03:30.173486 15559 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:03:30.173758 15551 server_base.cc:1034] running on GCE node
I20260619 09:03:30.173946 15551 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 09:03:30.174162 15551 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 09:03:30.175310 15551 hybrid_clock.cc:648] HybridClock initialized: now 1781859810175300 us; error 27 us; skew 500 ppm
Jun 19 09:03:30 dist-test-slave-mtdr krb5kdc[14113](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859810, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 09:03:30.177592 15551 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 09:03:30.178004 15551 webserver.cc:460] Webserver started at http://127.0.98.3:45903/ using document root <none> and password file <none>
I20260619 09:03:30.178213 15551 fs_manager.cc:359] Metadata directory not provided
I20260619 09:03:30.178260 15551 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 09:03:30.179497 15551 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.000s
I20260619 09:03:30.180007 15566 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 09:03:30.180171 15551 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260619 09:03:30.180246 15551 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859658134510-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859658134510-392-0/minicluster-data/ts-2/wal
uuid: "8a40e1510e3a47aab0fc73aa2a498625"
format_stamp: "Formatted at 2026-06-19 09:03:28 on dist-test-slave-mtdr"
server_key: "b48097f65193a9116d9d78149127f2e0"
server_key_iv: "22707887f15ccc2f18c1c79b23928b55"
server_key_version: "encryptionkey@0"
I20260619 09:03:30.180541 15551 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859658134510-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859658134510-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859658134510-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 09:03:30.213987 15551 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 09:03:30.215977 15551 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 09:03:30.216107 15551 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 09:03:30.216317 15551 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 09:03:30.216602 15551 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 09:03:30.216634 15551 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 09:03:30.216662 15551 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 09:03:30.216679 15551 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 09:03:30.221858 15551 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35365
I20260619 09:03:30.221917 15679 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35365 every 8 connection(s)
I20260619 09:03:30.222216 15551 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781859658134510-392-0/minicluster-data/ts-2/data/info.pb
I20260619 09:03:30.223140 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ez5NB/build/release/bin/kudu as pid 15551
I20260619 09:03:30.223242 15682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:03:30.222792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42391 (local address 127.0.98.3:39759)
0619 09:03:30.222967 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0619 09:03:30.223056 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":84,"threads_started":1}
W20260619 09:03:30.223642 15680 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:42391 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:42391: Client connection negotiation failed: client connection to 127.0.98.62:42391: connect: Connection refused (error 111)
I20260619 09:03:30.223898 15682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:03:30.223814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42391 (local address 127.0.98.3:38595)
0619 09:03:30.223853 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0619 09:03:30.223875 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260619 09:03:30.224135 15682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:03:30.224062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42391 (local address 127.0.98.3:42271)
0619 09:03:30.224097 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0619 09:03:30.224117 (+ 20us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
W20260619 09:03:30.224223 15680 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260619 09:03:30.999935 15684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:03:30.999703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42391 (local address 127.0.98.1:39289)
0619 09:03:30.999826 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0619 09:03:30.999871 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":59,"threads_started":1}
I20260619 09:03:31.084331 15685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:03:31.084119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42391 (local address 127.0.98.2:54545)
0619 09:03:31.084247 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0619 09:03:31.084286 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":59,"threads_started":1}
I20260619 09:03:31.206139 15686 raft_consensus.cc:482] T 5149ddddda2e48cebf0c32621d371916 P 2f05f67f28604e8faba8ecc8e8eb9569 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 09:03:31.206250 15686 raft_consensus.cc:504] T 5149ddddda2e48cebf0c32621d371916 P 2f05f67f28604e8faba8ecc8e8eb9569 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4857fc38e35489482ef9d9c80a69f4e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46005 } } peers { permanent_uuid: "8a40e1510e3a47aab0fc73aa2a498625" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35365 } } peers { permanent_uuid: "2f05f67f28604e8faba8ecc8e8eb9569" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33591 } }
I20260619 09:03:31.206544 15686 leader_election.cc:288] T 5149ddddda2e48cebf0c32621d371916 P 2f05f67f28604e8faba8ecc8e8eb9569 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a4857fc38e35489482ef9d9c80a69f4e (127.0.98.2:46005), 8a40e1510e3a47aab0fc73aa2a498625 (127.0.98.3:35365)
Jun 19 09:03:31 dist-test-slave-mtdr krb5kdc[14113](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859809, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.3@KRBTEST.COM
Jun 19 09:03:31 dist-test-slave-mtdr krb5kdc[14113](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859809, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
I20260619 09:03:31.216174 15688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:03:31.206877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35365 (local address 127.0.98.1:47583)
0619 09:03:31.206983 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0619 09:03:31.206993 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 09:03:31.207032 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:03:31.207826 (+ 794us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:03:31.207827 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:03:31.208013 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0619 09:03:31.208073 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:03:31.208075 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:03:31.209285 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:03:31.209286 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:03:31.209930 (+ 644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:03:31.209932 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:03:31.210479 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:03:31.210480 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:03:31.210577 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:03:31.211203 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 09:03:31.211209 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0619 09:03:31.212611 (+ 1402us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 09:03:31.214646 (+ 2035us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:03:31.214652 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:03:31.214653 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:03:31.214910 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:03:31.215178 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:03:31.215179 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:03:31.215179 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:03:31.215283 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:03:31.215700 (+ 417us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 09:03:31.215706 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 09:03:31.215818 (+ 112us) client_negotiation.cc:769] Sending connection context
0619 09:03:31.215901 (+ 83us) client_negotiation.cc:240] Negotiation successful
0619 09:03:31.216063 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":66,"threads_started":1}
I20260619 09:03:31.216187 15689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:03:31.206956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47583 (local address 127.0.98.3:35365)
0619 09:03:31.207109 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0619 09:03:31.207116 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 09:03:31.207135 (+ 19us) server_negotiation.cc:407] Connection header received
0619 09:03:31.207650 (+ 515us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:03:31.207652 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:03:31.207702 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:03:31.207827 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:03:31.208173 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:03:31.209197 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:03:31.210013 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:03:31.210404 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:03:31.210511 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:03:31.212712 (+ 2201us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:03:31.212715 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:03:31.212715 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:03:31.212724 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:03:31.214575 (+ 1851us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:03:31.214992 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:03:31.214996 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:03:31.214996 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:03:31.215096 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:03:31.215367 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:03:31.215368 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:03:31.215369 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:03:31.215630 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:03:31.215710 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0619 09:03:31.215974 (+ 264us) server_negotiation.cc:299] Negotiation successful
0619 09:03:31.216086 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":68,"threads_started":1}
I20260619 09:03:31.216418 15687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:03:31.206866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37399 (local address 127.0.98.2:46005)
0619 09:03:31.207050 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0619 09:03:31.207062 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 09:03:31.207090 (+ 28us) server_negotiation.cc:407] Connection header received
0619 09:03:31.207242 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:03:31.207245 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:03:31.207298 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:03:31.207438 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:03:31.208041 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:03:31.208997 (+ 956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:03:31.210010 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:03:31.210386 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:03:31.210489 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:03:31.213206 (+ 2717us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:03:31.213209 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:03:31.213209 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:03:31.213218 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:03:31.215239 (+ 2021us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:03:31.215506 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:03:31.215508 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:03:31.215509 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:03:31.215590 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:03:31.215751 (+ 161us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:03:31.215752 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:03:31.215752 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:03:31.215961 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:03:31.216027 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0619 09:03:31.216177 (+ 150us) server_negotiation.cc:299] Negotiation successful
0619 09:03:31.216296 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":58,"threads_started":1}
I20260619 09:03:31.216394 15684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:03:31.206719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46005 (local address 127.0.98.1:37399)
0619 09:03:31.206767 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0619 09:03:31.206780 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 09:03:31.206865 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:03:31.207471 (+ 606us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:03:31.207473 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:03:31.207750 (+ 277us) client_negotiation.cc:189] Negotiated authn=SASL
0619 09:03:31.207936 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:03:31.207942 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:03:31.209091 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:03:31.209092 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:03:31.209929 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:03:31.209931 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:03:31.210459 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:03:31.210460 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:03:31.210577 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:03:31.211152 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 09:03:31.211160 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 09:03:31.213112 (+ 1952us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 09:03:31.215299 (+ 2187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:03:31.215300 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:03:31.215301 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:03:31.215445 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:03:31.215644 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:03:31.215644 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:03:31.215645 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:03:31.215702 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:03:31.216020 (+ 318us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 09:03:31.216021 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 09:03:31.216060 (+ 39us) client_negotiation.cc:769] Sending connection context
0619 09:03:31.216130 (+ 70us) client_negotiation.cc:240] Negotiation successful
0619 09:03:31.216220 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27,"mutex_wait_us":4}
I20260619 09:03:31.217103 15501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5149ddddda2e48cebf0c32621d371916" candidate_uuid: "2f05f67f28604e8faba8ecc8e8eb9569" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "a4857fc38e35489482ef9d9c80a69f4e" is_pre_election: true
I20260619 09:03:31.217267 15501 raft_consensus.cc:2454] T 5149ddddda2e48cebf0c32621d371916 P a4857fc38e35489482ef9d9c80a69f4e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2f05f67f28604e8faba8ecc8e8eb9569 in term 1.
W20260619 09:03:31.217417 15300 leader_election.cc:341] T 5149ddddda2e48cebf0c32621d371916 P 2f05f67f28604e8faba8ecc8e8eb9569 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8a40e1510e3a47aab0fc73aa2a498625 (127.0.98.3:35365): Not found: Tablet not found: 5149ddddda2e48cebf0c32621d371916
I20260619 09:03:31.217510 15300 leader_election.cc:302] T 5149ddddda2e48cebf0c32621d371916 P 2f05f67f28604e8faba8ecc8e8eb9569 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2f05f67f28604e8faba8ecc8e8eb9569, a4857fc38e35489482ef9d9c80a69f4e; no voters: 8a40e1510e3a47aab0fc73aa2a498625
I20260619 09:03:31.217664 15686 raft_consensus.cc:2781] T 5149ddddda2e48cebf0c32621d371916 P 2f05f67f28604e8faba8ecc8e8eb9569 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260619 09:03:31.217720 15686 raft_consensus.cc:482] T 5149ddddda2e48cebf0c32621d371916 P 2f05f67f28604e8faba8ecc8e8eb9569 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 09:03:31.217742 15686 raft_consensus.cc:3037] T 5149ddddda2e48cebf0c32621d371916 P 2f05f67f28604e8faba8ecc8e8eb9569 [term 1 FOLLOWER]: Advancing to term 2
I20260619 09:03:31.218775 15686 raft_consensus.cc:504] T 5149ddddda2e48cebf0c32621d371916 P 2f05f67f28604e8faba8ecc8e8eb9569 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4857fc38e35489482ef9d9c80a69f4e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46005 } } peers { permanent_uuid: "8a40e1510e3a47aab0fc73aa2a498625" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35365 } } peers { permanent_uuid: "2f05f67f28604e8faba8ecc8e8eb9569" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33591 } }
I20260619 09:03:31.218912 15686 leader_election.cc:288] T 5149ddddda2e48cebf0c32621d371916 P 2f05f67f28604e8faba8ecc8e8eb9569 [CANDIDATE]: Term 2 election: Requested vote from peers a4857fc38e35489482ef9d9c80a69f4e (127.0.98.2:46005), 8a40e1510e3a47aab0fc73aa2a498625 (127.0.98.3:35365)
I20260619 09:03:31.219069 15501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5149ddddda2e48cebf0c32621d371916" candidate_uuid: "2f05f67f28604e8faba8ecc8e8eb9569" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "a4857fc38e35489482ef9d9c80a69f4e"
I20260619 09:03:31.219151 15501 raft_consensus.cc:3037] T 5149ddddda2e48cebf0c32621d371916 P a4857fc38e35489482ef9d9c80a69f4e [term 1 FOLLOWER]: Advancing to term 2
W20260619 09:03:31.219229 15300 leader_election.cc:341] T 5149ddddda2e48cebf0c32621d371916 P 2f05f67f28604e8faba8ecc8e8eb9569 [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer 8a40e1510e3a47aab0fc73aa2a498625 (127.0.98.3:35365): Not found: Tablet not found: 5149ddddda2e48cebf0c32621d371916
I20260619 09:03:31.220219 15501 raft_consensus.cc:2454] T 5149ddddda2e48cebf0c32621d371916 P a4857fc38e35489482ef9d9c80a69f4e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2f05f67f28604e8faba8ecc8e8eb9569 in term 2.
I20260619 09:03:31.220398 15300 leader_election.cc:302] T 5149ddddda2e48cebf0c32621d371916 P 2f05f67f28604e8faba8ecc8e8eb9569 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2f05f67f28604e8faba8ecc8e8eb9569, a4857fc38e35489482ef9d9c80a69f4e; no voters: 8a40e1510e3a47aab0fc73aa2a498625
I20260619 09:03:31.220474 15686 raft_consensus.cc:2781] T 5149ddddda2e48cebf0c32621d371916 P 2f05f67f28604e8faba8ecc8e8eb9569 [term 2 FOLLOWER]: Leader election won for term 2
I20260619 09:03:31.220569 15686 raft_consensus.cc:686] T 5149ddddda2e48cebf0c32621d371916 P 2f05f67f28604e8faba8ecc8e8eb9569 [term 2 LEADER]: Becoming Leader. State: Replica: 2f05f67f28604e8faba8ecc8e8eb9569, State: Running, Role: LEADER
I20260619 09:03:31.220664 15686 consensus_queue.cc:237] T 5149ddddda2e48cebf0c32621d371916 P 2f05f67f28604e8faba8ecc8e8eb9569 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4857fc38e35489482ef9d9c80a69f4e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46005 } } peers { permanent_uuid: "8a40e1510e3a47aab0fc73aa2a498625" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35365 } } peers { permanent_uuid: "2f05f67f28604e8faba8ecc8e8eb9569" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33591 } }
I20260619 09:03:31.220841 15684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:03:31.220744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42391 (local address 127.0.98.1:33993)
0619 09:03:31.220788 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0619 09:03:31.220818 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260619 09:03:31.224594 15692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:03:31.224448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42391 (local address 127.0.98.3:40653)
0619 09:03:31.224534 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0619 09:03:31.224566 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74,"thread_start_us":48,"threads_started":1}
W20260619 09:03:31.611565 15300 consensus_peers.cc:487] T 5149ddddda2e48cebf0c32621d371916 P 2f05f67f28604e8faba8ecc8e8eb9569 -> Peer 8a40e1510e3a47aab0fc73aa2a498625 (127.0.98.3:35365): Couldn't send request to peer 8a40e1510e3a47aab0fc73aa2a498625. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5149ddddda2e48cebf0c32621d371916. This is attempt 1: this message will repeat every 5th retry.
I20260619 09:03:31.769865 15501 raft_consensus.cc:1261] T 5149ddddda2e48cebf0c32621d371916 P a4857fc38e35489482ef9d9c80a69f4e [term 2 FOLLOWER]: Refusing update from remote peer 2f05f67f28604e8faba8ecc8e8eb9569: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260619 09:03:31.770215 15686 consensus_queue.cc:1035] T 5149ddddda2e48cebf0c32621d371916 P 2f05f67f28604e8faba8ecc8e8eb9569 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4857fc38e35489482ef9d9c80a69f4e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260619 09:03:31.771193 15695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:03:31.771031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42391 (local address 127.0.98.2:52619)
0619 09:03:31.771125 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0619 09:03:31.771157 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83,"thread_start_us":54,"threads_started":1}
I20260619 09:03:31.772109 15699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:03:31.771895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42391 (local address 127.0.98.1:39495)
0619 09:03:31.772041 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0619 09:03:31.772075 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":46,"threads_started":1}
I20260619 09:03:31.772184 15695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:03:31.772089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42391 (local address 127.0.98.2:59823)
0619 09:03:31.772130 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0619 09:03:31.772156 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260619 09:03:32.074924 15700 ts_tablet_manager.cc:927] T 5149ddddda2e48cebf0c32621d371916 P 8a40e1510e3a47aab0fc73aa2a498625: Initiating tablet copy from peer 2f05f67f28604e8faba8ecc8e8eb9569 (127.0.98.1:33591)
I20260619 09:03:32.075129 15700 tablet_copy_client.cc:316] T 5149ddddda2e48cebf0c32621d371916 P 8a40e1510e3a47aab0fc73aa2a498625: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.1:33591
Jun 19 09:03:32 dist-test-slave-mtdr krb5kdc[14113](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859810, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260619 09:03:32.082634 15702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:03:32.075454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33591 (local address 127.0.98.3:54737)
0619 09:03:32.075592 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0619 09:03:32.075609 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 09:03:32.075675 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:03:32.075857 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:03:32.075861 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:03:32.076149 (+ 288us) client_negotiation.cc:189] Negotiated authn=SASL
0619 09:03:32.076206 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:03:32.076208 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:03:32.077179 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:03:32.077180 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:03:32.077747 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:03:32.077748 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:03:32.078204 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:03:32.078205 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:03:32.078277 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:03:32.078718 (+ 441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 09:03:32.078722 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 09:03:32.080007 (+ 1285us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 09:03:32.081577 (+ 1570us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:03:32.081583 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:03:32.081586 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:03:32.081773 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:03:32.081969 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:03:32.081970 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:03:32.081970 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:03:32.082025 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:03:32.082270 (+ 245us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 09:03:32.082277 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 09:03:32.082330 (+ 53us) client_negotiation.cc:769] Sending connection context
0619 09:03:32.082405 (+ 75us) client_negotiation.cc:240] Negotiation successful
0619 09:03:32.082519 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":88,"threads_started":1}
I20260619 09:03:32.082645 15703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:03:32.075552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54737 (local address 127.0.98.1:33591)
0619 09:03:32.075724 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0619 09:03:32.075730 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 09:03:32.075739 (+ 9us) server_negotiation.cc:407] Connection header received
0619 09:03:32.075764 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:03:32.075766 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:03:32.075782 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:03:32.075845 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:03:32.076301 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:03:32.077094 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:03:32.077839 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:03:32.078140 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:03:32.078202 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:03:32.080084 (+ 1882us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:03:32.080092 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:03:32.080092 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:03:32.080099 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:03:32.081502 (+ 1403us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:03:32.081861 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:03:32.081863 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:03:32.081864 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:03:32.081915 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:03:32.082072 (+ 157us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:03:32.082073 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:03:32.082074 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:03:32.082211 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:03:32.082291 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0619 09:03:32.082443 (+ 152us) server_negotiation.cc:299] Negotiation successful
0619 09:03:32.082533 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":91,"threads_started":1}
I20260619 09:03:32.083214 15385 tablet_copy_service.cc:140] P 2f05f67f28604e8faba8ecc8e8eb9569: Received BeginTabletCopySession request for tablet 5149ddddda2e48cebf0c32621d371916 from peer 8a40e1510e3a47aab0fc73aa2a498625 ({username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:54737)
I20260619 09:03:32.083282 15385 tablet_copy_service.cc:161] P 2f05f67f28604e8faba8ecc8e8eb9569: Beginning new tablet copy session on tablet 5149ddddda2e48cebf0c32621d371916 from peer 8a40e1510e3a47aab0fc73aa2a498625 at {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:54737: session id = 8a40e1510e3a47aab0fc73aa2a498625-5149ddddda2e48cebf0c32621d371916
I20260619 09:03:32.083730 15385 tablet_copy_source_session.cc:205] T 5149ddddda2e48cebf0c32621d371916 P 2f05f67f28604e8faba8ecc8e8eb9569: Tablet Copy: opened 0 blocks and 1 log segments
I20260619 09:03:32.084653 15700 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5149ddddda2e48cebf0c32621d371916. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 09:03:32.087275 15700 tablet_copy_client.cc:791] T 5149ddddda2e48cebf0c32621d371916 P 8a40e1510e3a47aab0fc73aa2a498625: tablet copy: Starting download of 0 data blocks...
I20260619 09:03:32.087409 15700 tablet_copy_client.cc:655] T 5149ddddda2e48cebf0c32621d371916 P 8a40e1510e3a47aab0fc73aa2a498625: tablet copy: Starting download of 1 WAL segments...
I20260619 09:03:32.088061 15700 tablet_copy_client.cc:531] T 5149ddddda2e48cebf0c32621d371916 P 8a40e1510e3a47aab0fc73aa2a498625: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260619 09:03:32.088689 15700 tablet_bootstrap.cc:492] T 5149ddddda2e48cebf0c32621d371916 P 8a40e1510e3a47aab0fc73aa2a498625: Bootstrap starting.
I20260619 09:03:32.089910 15700 log.cc:824] T 5149ddddda2e48cebf0c32621d371916 P 8a40e1510e3a47aab0fc73aa2a498625: Log is configured to *not* fsync() on all Append() calls
I20260619 09:03:32.092231 15700 tablet_bootstrap.cc:492] T 5149ddddda2e48cebf0c32621d371916 P 8a40e1510e3a47aab0fc73aa2a498625: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 09:03:32.092418 15700 tablet_bootstrap.cc:492] T 5149ddddda2e48cebf0c32621d371916 P 8a40e1510e3a47aab0fc73aa2a498625: Bootstrap complete.
I20260619 09:03:32.092528 15700 ts_tablet_manager.cc:1397] T 5149ddddda2e48cebf0c32621d371916 P 8a40e1510e3a47aab0fc73aa2a498625: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260619 09:03:32.093423 15700 raft_consensus.cc:348] T 5149ddddda2e48cebf0c32621d371916 P 8a40e1510e3a47aab0fc73aa2a498625 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4857fc38e35489482ef9d9c80a69f4e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46005 } } peers { permanent_uuid: "8a40e1510e3a47aab0fc73aa2a498625" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35365 } } peers { permanent_uuid: "2f05f67f28604e8faba8ecc8e8eb9569" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33591 } }
I20260619 09:03:32.093556 15700 raft_consensus.cc:729] T 5149ddddda2e48cebf0c32621d371916 P 8a40e1510e3a47aab0fc73aa2a498625 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a40e1510e3a47aab0fc73aa2a498625, State: Initialized, Role: FOLLOWER
I20260619 09:03:32.093668 15700 consensus_queue.cc:260] T 5149ddddda2e48cebf0c32621d371916 P 8a40e1510e3a47aab0fc73aa2a498625 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4857fc38e35489482ef9d9c80a69f4e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46005 } } peers { permanent_uuid: "8a40e1510e3a47aab0fc73aa2a498625" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35365 } } peers { permanent_uuid: "2f05f67f28604e8faba8ecc8e8eb9569" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33591 } }
I20260619 09:03:32.093899 15700 ts_tablet_manager.cc:1428] T 5149ddddda2e48cebf0c32621d371916 P 8a40e1510e3a47aab0fc73aa2a498625: Time spent starting tablet: real 0.001s user 0.003s sys 0.000s
I20260619 09:03:32.094142 15385 tablet_copy_service.cc:342] P 2f05f67f28604e8faba8ecc8e8eb9569: Request end of tablet copy session 8a40e1510e3a47aab0fc73aa2a498625-5149ddddda2e48cebf0c32621d371916 received from {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:54737
I20260619 09:03:32.094172 15702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:03:32.094062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42391 (local address 127.0.98.3:45385)
0619 09:03:32.094112 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0619 09:03:32.094146 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260619 09:03:32.094198 15385 tablet_copy_service.cc:433] P 2f05f67f28604e8faba8ecc8e8eb9569: ending tablet copy session 8a40e1510e3a47aab0fc73aa2a498625-5149ddddda2e48cebf0c32621d371916 on tablet 5149ddddda2e48cebf0c32621d371916 with peer 8a40e1510e3a47aab0fc73aa2a498625
I20260619 09:03:32.094456 15702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:03:32.094376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42391 (local address 127.0.98.3:54803)
0619 09:03:32.094408 (+ 32us) negotiation.cc:107] Waiting for socket to connect
0619 09:03:32.094434 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
I20260619 09:03:32.558130 15707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:03:32.557935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42391 (local address 127.0.98.1:43481)
0619 09:03:32.558023 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0619 09:03:32.558079 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71,"thread_start_us":49,"threads_started":1}
I20260619 09:03:32.558091 15634 raft_consensus.cc:1203] T 5149ddddda2e48cebf0c32621d371916 P 8a40e1510e3a47aab0fc73aa2a498625 [term 2 FOLLOWER]: Deduplicated request from leader. Original: 1.2->[2.3-2.3] Dedup: 2.3->[]
I20260619 09:03:32.558634 15702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:03:32.558520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42391 (local address 127.0.98.3:44247)
0619 09:03:32.558570 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0619 09:03:32.558605 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260619 09:03:32.772785 15708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:03:32.772562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42391 (local address 127.0.98.2:49853)
0619 09:03:32.772695 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0619 09:03:32.772739 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":59,"threads_started":1}
I20260619 09:03:33.558861 15709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:03:33.558630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42391 (local address 127.0.98.1:37853)
0619 09:03:33.558744 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0619 09:03:33.558776 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":59,"threads_started":1}
I20260619 09:03:33.559152 15710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:03:33.558997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42391 (local address 127.0.98.3:58285)
0619 09:03:33.559083 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0619 09:03:33.559113 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70,"thread_start_us":48,"threads_started":1}
I20260619 09:03:33.773473 15711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:03:33.773235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42391 (local address 127.0.98.2:33661)
0619 09:03:33.773391 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0619 09:03:33.773427 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":75,"threads_started":1}
I20260619 09:03:34.559446 15712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:03:34.559241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42391 (local address 127.0.98.1:51511)
0619 09:03:34.559356 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0619 09:03:34.559389 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":71,"threads_started":1}
I20260619 09:03:34.559705 15713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:03:34.559507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42391 (local address 127.0.98.3:54665)
0619 09:03:34.559617 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0619 09:03:34.559655 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":63,"threads_started":1}
I20260619 09:03:34.774078 15714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:03:34.773874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42391 (local address 127.0.98.2:52289)
0619 09:03:34.774003 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0619 09:03:34.774032 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":73,"threads_started":1}
I20260619 09:03:35.560279 15716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:03:35.560113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42391 (local address 127.0.98.3:42367)
0619 09:03:35.560209 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0619 09:03:35.560239 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82,"thread_start_us":45,"threads_started":1}
I20260619 09:03:35.560271 15715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:03:35.559982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42391 (local address 127.0.98.1:39595)
0619 09:03:35.560169 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0619 09:03:35.560211 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":107,"threads_started":1}
I20260619 09:03:35.560645 15715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:03:35.560553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42391 (local address 127.0.98.1:41209)
0619 09:03:35.560597 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0619 09:03:35.560622 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260619 09:03:35.774869 15717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:03:35.774560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42391 (local address 127.0.98.2:41639)
0619 09:03:35.774729 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0619 09:03:35.774781 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":79,"threads_started":1}
I20260619 09:03:36.560874 15718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:03:36.560673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42391 (local address 127.0.98.3:47137)
0619 09:03:36.560785 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0619 09:03:36.560820 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":61,"threads_started":1}
I20260619 09:03:36.561184 15719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:03:36.561014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42391 (local address 127.0.98.1:48821)
0619 09:03:36.561105 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0619 09:03:36.561135 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76,"thread_start_us":53,"threads_started":1}
I20260619 09:03:36.775554 15720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:03:36.775348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42391 (local address 127.0.98.2:42951)
0619 09:03:36.775477 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0619 09:03:36.775507 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":66,"threads_started":1}
I20260619 09:03:37.561533 15721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:03:37.561292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42391 (local address 127.0.98.3:48163)
0619 09:03:37.561428 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0619 09:03:37.561475 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":60,"threads_started":1}
I20260619 09:03:37.561754 15722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:03:37.561560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42391 (local address 127.0.98.1:40963)
0619 09:03:37.561693 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0619 09:03:37.561721 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":58,"threads_started":1}
I20260619 09:03:37.776273 15723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:03:37.776074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42391 (local address 127.0.98.2:51051)
0619 09:03:37.776193 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0619 09:03:37.776227 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":63,"threads_started":1}
I20260619 09:03:38.562172 15724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:03:38.561959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42391 (local address 127.0.98.3:36211)
0619 09:03:38.562096 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0619 09:03:38.562126 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":67,"threads_started":1}
I20260619 09:03:38.562381 15725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:03:38.562150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42391 (local address 127.0.98.1:39863)
0619 09:03:38.562295 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0619 09:03:38.562331 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":62,"threads_started":1}
I20260619 09:03:38.776899 15726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:03:38.776701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42391 (local address 127.0.98.2:60471)
0619 09:03:38.776817 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0619 09:03:38.776853 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":57,"threads_started":1}
I20260619 09:03:39.562808 15727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:03:39.562596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42391 (local address 127.0.98.3:46991)
0619 09:03:39.562708 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0619 09:03:39.562762 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":58,"threads_started":1}
I20260619 09:03:39.563005 15728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:03:39.562791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42391 (local address 127.0.98.1:54547)
0619 09:03:39.562924 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0619 09:03:39.562958 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":86,"threads_started":1}
I20260619 09:03:39.777581 15729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:03:39.777345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42391 (local address 127.0.98.2:50247)
0619 09:03:39.777483 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0619 09:03:39.777523 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":60,"threads_started":1}
I20260619 09:03:40.563535 15731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:03:40.563362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42391 (local address 127.0.98.1:43497)
0619 09:03:40.563464 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0619 09:03:40.563491 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87,"thread_start_us":42,"threads_started":1}
I20260619 09:03:40.563529 15730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:03:40.563299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42391 (local address 127.0.98.3:52345)
0619 09:03:40.563440 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0619 09:03:40.563481 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":89,"threads_started":1}
I20260619 09:03:40.778245 15732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:03:40.778031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42391 (local address 127.0.98.2:37105)
0619 09:03:40.778163 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0619 09:03:40.778198 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":62,"threads_started":1}
I20260619 09:03:41.564206 15734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:03:41.563999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42391 (local address 127.0.98.1:59711)
0619 09:03:41.564132 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0619 09:03:41.564160 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":67,"threads_started":1}
I20260619 09:03:41.564247 15733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:03:41.564002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42391 (local address 127.0.98.3:48589)
0619 09:03:41.564161 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0619 09:03:41.564205 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":136,"threads_started":1}
I20260619 09:03:41.778987 15735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:03:41.778725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42391 (local address 127.0.98.2:55925)
0619 09:03:41.778904 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0619 09:03:41.778940 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":73,"threads_started":1}
I20260619 09:03:42.564862 15736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:03:42.564646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42391 (local address 127.0.98.1:40911)
0619 09:03:42.564782 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0619 09:03:42.564814 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":60,"threads_started":1}
I20260619 09:03:42.564885 15737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:03:42.564672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42391 (local address 127.0.98.3:52249)
0619 09:03:42.564791 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0619 09:03:42.564843 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":60,"threads_started":1}
I20260619 09:03:42.779726 15738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:03:42.779485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42391 (local address 127.0.98.2:59761)
0619 09:03:42.779635 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0619 09:03:42.779677 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":82,"threads_started":1}
I20260619 09:03:43.565461 15740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:03:43.565270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42391 (local address 127.0.98.3:57949)
0619 09:03:43.565393 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0619 09:03:43.565417 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":60,"threads_started":1}
I20260619 09:03:43.565470 15739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:03:43.565268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42391 (local address 127.0.98.1:49777)
0619 09:03:43.565383 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0619 09:03:43.565410 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":59,"threads_started":1}
I20260619 09:03:43.780452 15741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:03:43.780218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42391 (local address 127.0.98.2:34675)
0619 09:03:43.780361 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0619 09:03:43.780403 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":63,"threads_started":1}
I20260619 09:03:44.566210 15742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:03:44.565976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42391 (local address 127.0.98.3:53437)
0619 09:03:44.566108 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0619 09:03:44.566140 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":71,"threads_started":1}
I20260619 09:03:44.566193 15743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:03:44.565980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42391 (local address 127.0.98.1:59415)
0619 09:03:44.566106 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0619 09:03:44.566140 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":67,"threads_started":1}
I20260619 09:03:44.781216 15744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:03:44.780995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42391 (local address 127.0.98.2:47381)
0619 09:03:44.781127 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0619 09:03:44.781164 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":76,"threads_started":1}
I20260619 09:03:45.223560 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ez5NB/build/release/bin/kudu with pid 15252
I20260619 09:03:45.228104 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ez5NB/build/release/bin/kudu with pid 15414
I20260619 09:03:45.231922 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ez5NB/build/release/bin/kudu with pid 15551
2026-06-19T09:03:45Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (18022 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859658134510-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 09:03:45 dist-test-slave-mtdr krb5kdc[15747](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 09:03:45 dist-test-slave-mtdr krb5kdc[15747](info): set up 2 sockets
Jun 19 09:03:45 dist-test-slave-mtdr krb5kdc[15747](info): commencing operation
krb5kdc: starting...
W20260619 09:03:46.186563 392 mini_kdc.cc:121] Time spent starting KDC: real 0.935s user 0.004s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859658134510-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859658134510-392-0/krb5kdc/test-user.keytab.
Jun 19 09:03:46 dist-test-slave-mtdr krb5kdc[15747](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859826, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T09:03:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T09:03:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859658134510-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859658134510-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 09:03:46.297376 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ez5NB/build/release/bin/kudu
/tmp/dist-test-task4ez5NB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859658134510-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859658134510-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859658134510-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859658134510-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42887
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859658134510-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42887
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859658134510-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859658134510-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859658134510-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 09:03:46.376227 15760 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 09:03:46.376428 15760 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 09:03:46.376449 15760 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 09:03:46.377554 15760 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260619 09:03:46.377594 15760 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260619 09:03:46.377606 15760 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260619 09:03:46.377616 15760 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 09:03:46.377628 15760 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 09:03:46.377640 15760 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 09:03:46.377650 15760 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 09:03:46.377664 15760 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 09:03:46.379119 15760 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859658134510-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859658134510-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42887
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859658134510-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42887
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859658134510-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15760
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859658134510-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 09:03:46.379338 15760 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 09:03:46.379544 15760 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 09:03:46.384035 15765 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:03:46.384130 15766 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:03:46.384030 15768 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:03:46.384223 15760 server_base.cc:1034] running on GCE node
I20260619 09:03:46.384462 15760 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 09:03:46.384692 15760 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 09:03:46.385823 15760 hybrid_clock.cc:648] HybridClock initialized: now 1781859826385807 us; error 27 us; skew 500 ppm
Jun 19 09:03:46 dist-test-slave-mtdr krb5kdc[15747](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859826, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 09:03:46.388180 15760 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 09:03:46.388605 15760 webserver.cc:460] Webserver started at http://127.0.98.62:42285/ using document root <none> and password file <none>
I20260619 09:03:46.388795 15760 fs_manager.cc:359] Metadata directory not provided
I20260619 09:03:46.388842 15760 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 09:03:46.388960 15760 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 09:03:46.389932 15760 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859658134510-392-0/minicluster-data/master-0/data/instance:
uuid: "df0d2dbbe48b4c98a080b5e59d3c993d"
format_stamp: "Formatted at 2026-06-19 09:03:46 on dist-test-slave-mtdr"
server_key: "2de344460d2ca414a0dcf14bc1f4db7b"
server_key_iv: "8dfc905c00a66b785545414fb96c3160"
server_key_version: "encryptionkey@0"
I20260619 09:03:46.390255 15760 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859658134510-392-0/minicluster-data/master-0/wal/instance:
uuid: "df0d2dbbe48b4c98a080b5e59d3c993d"
format_stamp: "Formatted at 2026-06-19 09:03:46 on dist-test-slave-mtdr"
server_key: "2de344460d2ca414a0dcf14bc1f4db7b"
server_key_iv: "8dfc905c00a66b785545414fb96c3160"
server_key_version: "encryptionkey@0"
I20260619 09:03:46.391546 15760 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 09:03:46.392236 15775 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 09:03:46.392396 15760 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 09:03:46.392475 15760 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859658134510-392-0/minicluster-data/master-0/data,/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859658134510-392-0/minicluster-data/master-0/wal
uuid: "df0d2dbbe48b4c98a080b5e59d3c993d"
format_stamp: "Formatted at 2026-06-19 09:03:46 on dist-test-slave-mtdr"
server_key: "2de344460d2ca414a0dcf14bc1f4db7b"
server_key_iv: "8dfc905c00a66b785545414fb96c3160"
server_key_version: "encryptionkey@0"
I20260619 09:03:46.392534 15760 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859658134510-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859658134510-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859658134510-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 09:03:46.404978 15760 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 09:03:46.406718 15760 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 09:03:46.406867 15760 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 09:03:46.410506 15760 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42887
I20260619 09:03:46.410537 15827 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42887 every 8 connection(s)
I20260619 09:03:46.410909 15760 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859658134510-392-0/minicluster-data/master-0/data/info.pb
I20260619 09:03:46.411679 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ez5NB/build/release/bin/kudu as pid 15760
I20260619 09:03:46.411784 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859658134510-392-0/minicluster-data/master-0/wal/instance
I20260619 09:03:46.411782 15828 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 09:03:46.412002 392 external_mini_cluster.cc:1428] Setting key 07c96e6c27068e3e8af6db61ebdef151
I20260619 09:03:46.414666 15828 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df0d2dbbe48b4c98a080b5e59d3c993d: Bootstrap starting.
I20260619 09:03:46.415331 15828 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P df0d2dbbe48b4c98a080b5e59d3c993d: Neither blocks nor log segments found. Creating new log.
I20260619 09:03:46.415619 15828 log.cc:824] T 00000000000000000000000000000000 P df0d2dbbe48b4c98a080b5e59d3c993d: Log is configured to *not* fsync() on all Append() calls
I20260619 09:03:46.416260 15828 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df0d2dbbe48b4c98a080b5e59d3c993d: No bootstrap required, opened a new log
I20260619 09:03:46.418232 15828 raft_consensus.cc:348] T 00000000000000000000000000000000 P df0d2dbbe48b4c98a080b5e59d3c993d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df0d2dbbe48b4c98a080b5e59d3c993d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42887 } }
I20260619 09:03:46.418393 15828 raft_consensus.cc:374] T 00000000000000000000000000000000 P df0d2dbbe48b4c98a080b5e59d3c993d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 09:03:46.418427 15828 raft_consensus.cc:729] T 00000000000000000000000000000000 P df0d2dbbe48b4c98a080b5e59d3c993d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df0d2dbbe48b4c98a080b5e59d3c993d, State: Initialized, Role: FOLLOWER
Jun 19 09:03:46 dist-test-slave-mtdr krb5kdc[15747](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859826, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 09:03:46.418558 15828 consensus_queue.cc:260] T 00000000000000000000000000000000 P df0d2dbbe48b4c98a080b5e59d3c993d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df0d2dbbe48b4c98a080b5e59d3c993d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42887 } }
I20260619 09:03:46.418638 15828 raft_consensus.cc:388] T 00000000000000000000000000000000 P df0d2dbbe48b4c98a080b5e59d3c993d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 09:03:46.418669 15828 raft_consensus.cc:482] T 00000000000000000000000000000000 P df0d2dbbe48b4c98a080b5e59d3c993d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 09:03:46.418717 15828 raft_consensus.cc:3037] T 00000000000000000000000000000000 P df0d2dbbe48b4c98a080b5e59d3c993d [term 0 FOLLOWER]: Advancing to term 1
I20260619 09:03:46.419368 15828 raft_consensus.cc:504] T 00000000000000000000000000000000 P df0d2dbbe48b4c98a080b5e59d3c993d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df0d2dbbe48b4c98a080b5e59d3c993d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42887 } }
I20260619 09:03:46.419466 15828 leader_election.cc:302] T 00000000000000000000000000000000 P df0d2dbbe48b4c98a080b5e59d3c993d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: df0d2dbbe48b4c98a080b5e59d3c993d; no voters:
I20260619 09:03:46.419628 15828 leader_election.cc:288] T 00000000000000000000000000000000 P df0d2dbbe48b4c98a080b5e59d3c993d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 09:03:46.419694 15833 raft_consensus.cc:2781] T 00000000000000000000000000000000 P df0d2dbbe48b4c98a080b5e59d3c993d [term 1 FOLLOWER]: Leader election won for term 1
I20260619 09:03:46.419875 15828 sys_catalog.cc:564] T 00000000000000000000000000000000 P df0d2dbbe48b4c98a080b5e59d3c993d [sys.catalog]: configured and running, proceeding with master startup.
I20260619 09:03:46.419873 15833 raft_consensus.cc:686] T 00000000000000000000000000000000 P df0d2dbbe48b4c98a080b5e59d3c993d [term 1 LEADER]: Becoming Leader. State: Replica: df0d2dbbe48b4c98a080b5e59d3c993d, State: Running, Role: LEADER
I20260619 09:03:46.421092 15833 consensus_queue.cc:237] T 00000000000000000000000000000000 P df0d2dbbe48b4c98a080b5e59d3c993d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df0d2dbbe48b4c98a080b5e59d3c993d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42887 } }
I20260619 09:03:46.421708 15835 sys_catalog.cc:455] T 00000000000000000000000000000000 P df0d2dbbe48b4c98a080b5e59d3c993d [sys.catalog]: SysCatalogTable state changed. Reason: New leader df0d2dbbe48b4c98a080b5e59d3c993d. Latest consensus state: current_term: 1 leader_uuid: "df0d2dbbe48b4c98a080b5e59d3c993d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df0d2dbbe48b4c98a080b5e59d3c993d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42887 } } }
I20260619 09:03:46.421926 15835 sys_catalog.cc:458] T 00000000000000000000000000000000 P df0d2dbbe48b4c98a080b5e59d3c993d [sys.catalog]: This master's current role is: LEADER
I20260619 09:03:46.421734 15834 sys_catalog.cc:455] T 00000000000000000000000000000000 P df0d2dbbe48b4c98a080b5e59d3c993d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "df0d2dbbe48b4c98a080b5e59d3c993d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df0d2dbbe48b4c98a080b5e59d3c993d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42887 } } }
I20260619 09:03:46.422106 15834 sys_catalog.cc:458] T 00000000000000000000000000000000 P df0d2dbbe48b4c98a080b5e59d3c993d [sys.catalog]: This master's current role is: LEADER
I20260619 09:03:46.422120 15849 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 09:03:46.422638 15849 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 09:03:46.422868 15831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:03:46.412495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49308 (local address 127.0.98.62:42887)
0619 09:03:46.412647 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0619 09:03:46.412654 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 09:03:46.412678 (+ 24us) server_negotiation.cc:407] Connection header received
0619 09:03:46.413117 (+ 439us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:03:46.413123 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:03:46.413167 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:03:46.413273 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:03:46.414003 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:03:46.414966 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:03:46.415974 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:03:46.416413 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:03:46.416507 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:03:46.419033 (+ 2526us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:03:46.419037 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:03:46.419037 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:03:46.419048 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:03:46.420934 (+ 1886us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:03:46.421317 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:03:46.421320 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:03:46.421320 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:03:46.421420 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:03:46.421938 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:03:46.421939 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:03:46.421940 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:03:46.422240 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:03:46.422357 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0619 09:03:46.422527 (+ 170us) server_negotiation.cc:299] Negotiation successful
0619 09:03:46.422658 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":64,"threads_started":1}
I20260619 09:03:46.424166 15849 catalog_manager.cc:1347] Generated new cluster ID: fb77a7cd21ee4af78f446a5fba7eb31e
I20260619 09:03:46.424207 15849 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 09:03:46.442198 15849 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 09:03:46.442848 15849 catalog_manager.cc:1504] Loading token signing keys...
I20260619 09:03:46.452016 15849 catalog_manager.cc:5868] T 00000000000000000000000000000000 P df0d2dbbe48b4c98a080b5e59d3c993d: Generated new TSK 0
I20260619 09:03:46.452282 15849 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859658134510-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859658134510-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 09:03:46.482113 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ez5NB/build/release/bin/kudu
/tmp/dist-test-task4ez5NB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859658134510-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859658134510-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859658134510-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859658134510-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42887
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859658134510-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859658134510-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859658134510-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859658134510-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 09:03:46.559654 15854 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 09:03:46.559849 15854 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 09:03:46.559870 15854 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 09:03:46.560972 15854 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 09:03:46.561008 15854 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 09:03:46.561041 15854 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 09:03:46.562408 15854 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859658134510-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859658134510-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859658134510-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859658134510-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42887
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15854
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859658134510-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 09:03:46.562584 15854 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 09:03:46.562752 15854 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 09:03:46.567266 15860 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:03:46.567330 15862 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:03:46.567413 15859 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:03:46.567646 15854 server_base.cc:1034] running on GCE node
I20260619 09:03:46.567822 15854 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 09:03:46.568043 15854 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 09:03:46.569190 15854 hybrid_clock.cc:648] HybridClock initialized: now 1781859826569175 us; error 29 us; skew 500 ppm
Jun 19 09:03:46 dist-test-slave-mtdr krb5kdc[15747](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859826, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 09:03:46.571450 15854 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 09:03:46.571830 15854 webserver.cc:460] Webserver started at http://127.0.98.1:34623/ using document root <none> and password file <none>
I20260619 09:03:46.572011 15854 fs_manager.cc:359] Metadata directory not provided
I20260619 09:03:46.572057 15854 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 09:03:46.572175 15854 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 09:03:46.573114 15854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859658134510-392-0/minicluster-data/ts-0/data/instance:
uuid: "c9252303f6c147b99263a5c7666a6134"
format_stamp: "Formatted at 2026-06-19 09:03:46 on dist-test-slave-mtdr"
server_key: "4c23a7631a8101e8f8f522d0837b3dc8"
server_key_iv: "ff9c47905da211813b68b95dda196455"
server_key_version: "encryptionkey@0"
I20260619 09:03:46.573431 15854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859658134510-392-0/minicluster-data/ts-0/wal/instance:
uuid: "c9252303f6c147b99263a5c7666a6134"
format_stamp: "Formatted at 2026-06-19 09:03:46 on dist-test-slave-mtdr"
server_key: "4c23a7631a8101e8f8f522d0837b3dc8"
server_key_iv: "ff9c47905da211813b68b95dda196455"
server_key_version: "encryptionkey@0"
I20260619 09:03:46.574559 15854 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 09:03:46.575222 15869 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 09:03:46.575369 15854 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260619 09:03:46.575446 15854 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859658134510-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859658134510-392-0/minicluster-data/ts-0/wal
uuid: "c9252303f6c147b99263a5c7666a6134"
format_stamp: "Formatted at 2026-06-19 09:03:46 on dist-test-slave-mtdr"
server_key: "4c23a7631a8101e8f8f522d0837b3dc8"
server_key_iv: "ff9c47905da211813b68b95dda196455"
server_key_version: "encryptionkey@0"
I20260619 09:03:46.575510 15854 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859658134510-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859658134510-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859658134510-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 09:03:46.599964 15854 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 09:03:46.601701 15854 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 09:03:46.601833 15854 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 09:03:46.602046 15854 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 09:03:46.602356 15854 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 09:03:46.602391 15854 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 09:03:46.602419 15854 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 09:03:46.602437 15854 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 09:03:46.608044 15854 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41245
I20260619 09:03:46.608115 15982 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41245 every 8 connection(s)
I20260619 09:03:46.608390 15854 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859658134510-392-0/minicluster-data/ts-0/data/info.pb
Jun 19 09:03:46 dist-test-slave-mtdr krb5kdc[15747](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859826, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 09:03:46.616576 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ez5NB/build/release/bin/kudu as pid 15854
I20260619 09:03:46.616619 15831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:03:46.609016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45631 (local address 127.0.98.62:42887)
0619 09:03:46.609094 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0619 09:03:46.609097 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 09:03:46.609265 (+ 168us) server_negotiation.cc:407] Connection header received
0619 09:03:46.609759 (+ 494us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:03:46.609760 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:03:46.609770 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:03:46.609810 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:03:46.610676 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:03:46.611226 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:03:46.612212 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:03:46.612399 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:03:46.612440 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:03:46.614313 (+ 1873us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:03:46.614314 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:03:46.614315 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:03:46.614321 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:03:46.615603 (+ 1282us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:03:46.615915 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:03:46.615916 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:03:46.615916 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:03:46.615960 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:03:46.616130 (+ 170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:03:46.616131 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:03:46.616131 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:03:46.616226 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:03:46.616272 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0619 09:03:46.616485 (+ 213us) server_negotiation.cc:299] Negotiation successful
0619 09:03:46.616545 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 09:03:46.616679 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859658134510-392-0/minicluster-data/ts-0/wal/instance
I20260619 09:03:46.616864 15985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:03:46.608986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42887 (local address 127.0.98.1:45631)
0619 09:03:46.609142 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0619 09:03:46.609178 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0619 09:03:46.609669 (+ 491us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:03:46.609867 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:03:46.609870 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:03:46.610411 (+ 541us) client_negotiation.cc:189] Negotiated authn=SASL
0619 09:03:46.610607 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:03:46.610612 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:03:46.611319 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:03:46.611320 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:03:46.612146 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:03:46.612148 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:03:46.612451 (+ 303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:03:46.612452 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:03:46.612555 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:03:46.612966 (+ 411us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 09:03:46.612973 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0619 09:03:46.614233 (+ 1260us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 09:03:46.615665 (+ 1432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:03:46.615669 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:03:46.615671 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:03:46.615865 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:03:46.616002 (+ 137us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:03:46.616003 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:03:46.616003 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:03:46.616089 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:03:46.616267 (+ 178us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 09:03:46.616270 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 09:03:46.616374 (+ 104us) client_negotiation.cc:769] Sending connection context
0619 09:03:46.616474 (+ 100us) client_negotiation.cc:240] Negotiation successful
0619 09:03:46.616648 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":91,"threads_started":1}
I20260619 09:03:46.616894 392 external_mini_cluster.cc:1428] Setting key 66098d4930ab2bc2d2df08faa95117e2
I20260619 09:03:46.617434 15983 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42887
I20260619 09:03:46.617523 15983 heartbeater.cc:464] Registering TS with master...
I20260619 09:03:46.617702 15983 heartbeater.cc:511] Master 127.0.98.62:42887 requested a full tablet report, sending...
I20260619 09:03:46.618284 15792 ts_manager.cc:194] Registered new tserver with Master: c9252303f6c147b99263a5c7666a6134 (127.0.98.1:41245)
I20260619 09:03:46.619102 15792 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:45631
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859658134510-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859658134510-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 09:03:46.639483 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ez5NB/build/release/bin/kudu
/tmp/dist-test-task4ez5NB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859658134510-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859658134510-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859658134510-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859658134510-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42887
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859658134510-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859658134510-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859658134510-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859658134510-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 09:03:46.716938 15988 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 09:03:46.717124 15988 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 09:03:46.717147 15988 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 09:03:46.718206 15988 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 09:03:46.718240 15988 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 09:03:46.718264 15988 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 09:03:46.719646 15988 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859658134510-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859658134510-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859658134510-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859658134510-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42887
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15988
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859658134510-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 09:03:46.719821 15988 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 09:03:46.720011 15988 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 09:03:46.724941 15988 server_base.cc:1034] running on GCE node
W20260619 09:03:46.724869 15994 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:03:46.724888 15996 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:03:46.724906 15993 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:03:46.725329 15988 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 09:03:46.725569 15988 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 09:03:46.726707 15988 hybrid_clock.cc:648] HybridClock initialized: now 1781859826726696 us; error 24 us; skew 500 ppm
Jun 19 09:03:46 dist-test-slave-mtdr krb5kdc[15747](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859826, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 09:03:46.729023 15988 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 09:03:46.729434 15988 webserver.cc:460] Webserver started at http://127.0.98.2:36909/ using document root <none> and password file <none>
I20260619 09:03:46.729645 15988 fs_manager.cc:359] Metadata directory not provided
I20260619 09:03:46.729692 15988 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 09:03:46.729807 15988 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 09:03:46.730775 15988 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859658134510-392-0/minicluster-data/ts-1/data/instance:
uuid: "51ec1359afd84648a293aff68cab2a8e"
format_stamp: "Formatted at 2026-06-19 09:03:46 on dist-test-slave-mtdr"
server_key: "3c6eeecd77c8caf9312c2c16884bba17"
server_key_iv: "5bce3236820d4114c2c8a452b1ee9625"
server_key_version: "encryptionkey@0"
I20260619 09:03:46.731132 15988 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859658134510-392-0/minicluster-data/ts-1/wal/instance:
uuid: "51ec1359afd84648a293aff68cab2a8e"
format_stamp: "Formatted at 2026-06-19 09:03:46 on dist-test-slave-mtdr"
server_key: "3c6eeecd77c8caf9312c2c16884bba17"
server_key_iv: "5bce3236820d4114c2c8a452b1ee9625"
server_key_version: "encryptionkey@0"
I20260619 09:03:46.732285 15988 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 09:03:46.732893 16003 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 09:03:46.733045 15988 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 09:03:46.733135 15988 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859658134510-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859658134510-392-0/minicluster-data/ts-1/wal
uuid: "51ec1359afd84648a293aff68cab2a8e"
format_stamp: "Formatted at 2026-06-19 09:03:46 on dist-test-slave-mtdr"
server_key: "3c6eeecd77c8caf9312c2c16884bba17"
server_key_iv: "5bce3236820d4114c2c8a452b1ee9625"
server_key_version: "encryptionkey@0"
I20260619 09:03:46.733201 15988 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859658134510-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859658134510-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859658134510-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 09:03:46.741185 15988 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 09:03:46.743144 15988 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 09:03:46.743266 15988 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 09:03:46.743450 15988 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 09:03:46.743746 15988 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 09:03:46.743783 15988 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 09:03:46.743810 15988 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 09:03:46.743826 15988 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 09:03:46.749111 15988 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33867
I20260619 09:03:46.749168 16116 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33867 every 8 connection(s)
I20260619 09:03:46.749464 15988 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859658134510-392-0/minicluster-data/ts-1/data/info.pb
I20260619 09:03:46.753552 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ez5NB/build/release/bin/kudu as pid 15988
I20260619 09:03:46.753639 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859658134510-392-0/minicluster-data/ts-1/wal/instance
I20260619 09:03:46.753842 392 external_mini_cluster.cc:1428] Setting key 1644c4e75de2e0d31b06063ca261903d
Jun 19 09:03:46 dist-test-slave-mtdr krb5kdc[15747](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859826, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 09:03:46.758265 15831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:03:46.750154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54249 (local address 127.0.98.62:42887)
0619 09:03:46.750212 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0619 09:03:46.750214 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:03:46.750382 (+ 168us) server_negotiation.cc:407] Connection header received
0619 09:03:46.750858 (+ 476us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:03:46.750859 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:03:46.750871 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:03:46.750908 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:03:46.751793 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:03:46.752282 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:03:46.753230 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:03:46.753414 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:03:46.753467 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:03:46.755749 (+ 2282us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:03:46.755750 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:03:46.755750 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:03:46.755758 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:03:46.757024 (+ 1266us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:03:46.757433 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:03:46.757433 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:03:46.757434 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:03:46.757476 (+ 42us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:03:46.757707 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:03:46.757707 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:03:46.757708 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:03:46.757802 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:03:46.757846 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0619 09:03:46.758143 (+ 297us) server_negotiation.cc:299] Negotiation successful
0619 09:03:46.758201 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260619 09:03:46.758510 16119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:03:46.750107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42887 (local address 127.0.98.2:54249)
0619 09:03:46.750272 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0619 09:03:46.750307 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0619 09:03:46.750754 (+ 447us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:03:46.750985 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:03:46.750989 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:03:46.751524 (+ 535us) client_negotiation.cc:189] Negotiated authn=SASL
0619 09:03:46.751701 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:03:46.751705 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:03:46.752361 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:03:46.752362 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:03:46.753148 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:03:46.753150 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:03:46.753479 (+ 329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:03:46.753480 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:03:46.753596 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:03:46.754121 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 09:03:46.754131 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 09:03:46.755662 (+ 1531us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 09:03:46.757103 (+ 1441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:03:46.757109 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:03:46.757111 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:03:46.757370 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:03:46.757531 (+ 161us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:03:46.757532 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:03:46.757532 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:03:46.757651 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:03:46.757852 (+ 201us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 09:03:46.757857 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 09:03:46.758010 (+ 153us) client_negotiation.cc:769] Sending connection context
0619 09:03:46.758128 (+ 118us) client_negotiation.cc:240] Negotiation successful
0619 09:03:46.758285 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":89,"threads_started":1}
I20260619 09:03:46.758985 16117 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42887
I20260619 09:03:46.759081 16117 heartbeater.cc:464] Registering TS with master...
I20260619 09:03:46.759269 16117 heartbeater.cc:511] Master 127.0.98.62:42887 requested a full tablet report, sending...
I20260619 09:03:46.759618 15792 ts_manager.cc:194] Registered new tserver with Master: 51ec1359afd84648a293aff68cab2a8e (127.0.98.2:33867)
I20260619 09:03:46.759961 15792 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:54249
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859658134510-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859658134510-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 09:03:46.776832 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ez5NB/build/release/bin/kudu
/tmp/dist-test-task4ez5NB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859658134510-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859658134510-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859658134510-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859658134510-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42887
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859658134510-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859658134510-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859658134510-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859658134510-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 09:03:46.853693 16122 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 09:03:46.853876 16122 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 09:03:46.853899 16122 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 09:03:46.855005 16122 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 09:03:46.855041 16122 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 09:03:46.855069 16122 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 09:03:46.856467 16122 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859658134510-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859658134510-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859658134510-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859658134510-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42887
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16122
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859658134510-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 09:03:46.856676 16122 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 09:03:46.856868 16122 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 09:03:46.861609 16127 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:03:46.861696 16128 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:03:46.861646 16130 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:03:46.861900 16122 server_base.cc:1034] running on GCE node
I20260619 09:03:46.862083 16122 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 09:03:46.862282 16122 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 09:03:46.863404 16122 hybrid_clock.cc:648] HybridClock initialized: now 1781859826863395 us; error 24 us; skew 500 ppm
Jun 19 09:03:46 dist-test-slave-mtdr krb5kdc[15747](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859826, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 09:03:46.865584 16122 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 09:03:46.865962 16122 webserver.cc:460] Webserver started at http://127.0.98.3:38789/ using document root <none> and password file <none>
I20260619 09:03:46.866158 16122 fs_manager.cc:359] Metadata directory not provided
I20260619 09:03:46.866204 16122 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 09:03:46.866325 16122 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 09:03:46.867326 16122 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859658134510-392-0/minicluster-data/ts-2/data/instance:
uuid: "ce39db3fe58a4151a4795ecaf0942bc9"
format_stamp: "Formatted at 2026-06-19 09:03:46 on dist-test-slave-mtdr"
server_key: "24578ab589f048f7a70103b4c4f8d7cd"
server_key_iv: "ef05717263c0ad09e7d552c0c4058654"
server_key_version: "encryptionkey@0"
I20260619 09:03:46.867640 16122 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859658134510-392-0/minicluster-data/ts-2/wal/instance:
uuid: "ce39db3fe58a4151a4795ecaf0942bc9"
format_stamp: "Formatted at 2026-06-19 09:03:46 on dist-test-slave-mtdr"
server_key: "24578ab589f048f7a70103b4c4f8d7cd"
server_key_iv: "ef05717263c0ad09e7d552c0c4058654"
server_key_version: "encryptionkey@0"
I20260619 09:03:46.868783 16122 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 09:03:46.869396 16137 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 09:03:46.869541 16122 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 09:03:46.869617 16122 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859658134510-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859658134510-392-0/minicluster-data/ts-2/wal
uuid: "ce39db3fe58a4151a4795ecaf0942bc9"
format_stamp: "Formatted at 2026-06-19 09:03:46 on dist-test-slave-mtdr"
server_key: "24578ab589f048f7a70103b4c4f8d7cd"
server_key_iv: "ef05717263c0ad09e7d552c0c4058654"
server_key_version: "encryptionkey@0"
I20260619 09:03:46.869675 16122 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859658134510-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859658134510-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859658134510-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 09:03:46.879187 16122 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 09:03:46.880832 16122 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 09:03:46.880950 16122 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 09:03:46.881138 16122 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 09:03:46.881412 16122 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 09:03:46.881449 16122 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 09:03:46.881476 16122 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 09:03:46.881500 16122 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 09:03:46.886754 16122 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:46739
I20260619 09:03:46.886803 16250 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:46739 every 8 connection(s)
I20260619 09:03:46.887094 16122 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859658134510-392-0/minicluster-data/ts-2/data/info.pb
I20260619 09:03:46.890982 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ez5NB/build/release/bin/kudu as pid 16122
I20260619 09:03:46.891078 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859658134510-392-0/minicluster-data/ts-2/wal/instance
I20260619 09:03:46.891289 392 external_mini_cluster.cc:1428] Setting key 0e7da09fa3da62dd8d2b299eeed2fde7
Jun 19 09:03:46 dist-test-slave-mtdr krb5kdc[15747](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859826, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 09:03:46.895313 15831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:03:46.887704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42171 (local address 127.0.98.62:42887)
0619 09:03:46.887769 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0619 09:03:46.887771 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:03:46.887937 (+ 166us) server_negotiation.cc:407] Connection header received
0619 09:03:46.888390 (+ 453us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:03:46.888391 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:03:46.888400 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:03:46.888452 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:03:46.889298 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:03:46.889789 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:03:46.890685 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:03:46.890911 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:03:46.890959 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:03:46.893165 (+ 2206us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:03:46.893167 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:03:46.893167 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:03:46.893173 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:03:46.894203 (+ 1030us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:03:46.894573 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:03:46.894574 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:03:46.894574 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:03:46.894616 (+ 42us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:03:46.894837 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:03:46.894837 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:03:46.894837 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:03:46.894937 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:03:46.894986 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0619 09:03:46.895195 (+ 209us) server_negotiation.cc:299] Negotiation successful
0619 09:03:46.895250 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260619 09:03:46.895485 16253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:03:46.887657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42887 (local address 127.0.98.3:42171)
0619 09:03:46.887823 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0619 09:03:46.887862 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 09:03:46.888310 (+ 448us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:03:46.888512 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:03:46.888515 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:03:46.889051 (+ 536us) client_negotiation.cc:189] Negotiated authn=SASL
0619 09:03:46.889234 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:03:46.889238 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:03:46.889852 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:03:46.889853 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:03:46.890625 (+ 772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:03:46.890627 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:03:46.890990 (+ 363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:03:46.890991 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:03:46.891130 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:03:46.891645 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 09:03:46.891655 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 09:03:46.893089 (+ 1434us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 09:03:46.894271 (+ 1182us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:03:46.894277 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:03:46.894279 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:03:46.894519 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:03:46.894662 (+ 143us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:03:46.894663 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:03:46.894663 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:03:46.894752 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:03:46.894983 (+ 231us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 09:03:46.894988 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 09:03:46.895093 (+ 105us) client_negotiation.cc:769] Sending connection context
0619 09:03:46.895186 (+ 93us) client_negotiation.cc:240] Negotiation successful
0619 09:03:46.895302 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":98,"threads_started":1}
I20260619 09:03:46.895886 16251 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42887
I20260619 09:03:46.895983 16251 heartbeater.cc:464] Registering TS with master...
I20260619 09:03:46.896118 16251 heartbeater.cc:511] Master 127.0.98.62:42887 requested a full tablet report, sending...
I20260619 09:03:46.896445 15792 ts_manager.cc:194] Registered new tserver with Master: ce39db3fe58a4151a4795ecaf0942bc9 (127.0.98.3:46739)
I20260619 09:03:46.896799 15792 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:42171
I20260619 09:03:46.902983 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 09:03:46.909250 15831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:03:46.904279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49318 (local address 127.0.98.62:42887)
0619 09:03:46.904336 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0619 09:03:46.904338 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:03:46.904422 (+ 84us) server_negotiation.cc:407] Connection header received
0619 09:03:46.904480 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:03:46.904481 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:03:46.904489 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:03:46.904526 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:03:46.905027 (+ 501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:03:46.905484 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:03:46.906161 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:03:46.906340 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:03:46.906390 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:03:46.907432 (+ 1042us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:03:46.907433 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:03:46.907434 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:03:46.907439 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:03:46.908460 (+ 1021us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:03:46.908720 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:03:46.908720 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:03:46.908721 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:03:46.908761 (+ 40us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:03:46.908882 (+ 121us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:03:46.908882 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:03:46.908882 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:03:46.908996 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:03:46.909061 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0619 09:03:46.909138 (+ 77us) server_negotiation.cc:299] Negotiation successful
0619 09:03:46.909188 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260619 09:03:46.910543 15792 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49318:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 09:03:46.912065 15792 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 09:03:46.917917 16261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:03:46.914018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46739 (local address 127.0.0.1:55060)
0619 09:03:46.914156 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0619 09:03:46.914194 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 09:03:46.914259 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:03:46.914612 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:03:46.914614 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:03:46.914615 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 09:03:46.914673 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:03:46.914675 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:03:46.915829 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:03:46.915830 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:03:46.916864 (+ 1034us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:03:46.916866 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:03:46.917631 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:03:46.917632 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:03:46.917703 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:03:46.917716 (+ 13us) client_negotiation.cc:769] Sending connection context
0619 09:03:46.917787 (+ 71us) client_negotiation.cc:240] Negotiation successful
0619 09:03:46.917840 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"mutex_wait_us":53,"thread_start_us":80,"threads_started":1}
I20260619 09:03:46.917927 16263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:03:46.914283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55060 (local address 127.0.98.3:46739)
0619 09:03:46.914445 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0619 09:03:46.914448 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 09:03:46.914457 (+ 9us) server_negotiation.cc:407] Connection header received
0619 09:03:46.914492 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:03:46.914494 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:03:46.914509 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:03:46.914614 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 09:03:46.914757 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:03:46.915730 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:03:46.916946 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:03:46.917551 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:03:46.917621 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:03:46.917675 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0619 09:03:46.917805 (+ 130us) server_negotiation.cc:299] Negotiation successful
0619 09:03:46.917848 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":66,"threads_started":1}
I20260619 09:03:46.918274 16260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:03:46.914022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41245 (local address 127.0.0.1:42600)
0619 09:03:46.914364 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0619 09:03:46.914372 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0619 09:03:46.914407 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:03:46.914908 (+ 501us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:03:46.914909 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:03:46.914910 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 09:03:46.914944 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:03:46.914946 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:03:46.916151 (+ 1205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:03:46.916152 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:03:46.917212 (+ 1060us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:03:46.917214 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:03:46.918078 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:03:46.918079 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:03:46.918135 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:03:46.918139 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 09:03:46.918178 (+ 39us) client_negotiation.cc:240] Negotiation successful
0619 09:03:46.918213 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":83,"threads_started":1}
I20260619 09:03:46.918401 16259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:03:46.914018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33867 (local address 127.0.0.1:38536)
0619 09:03:46.914465 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0619 09:03:46.914472 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0619 09:03:46.914511 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:03:46.914658 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:03:46.914659 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:03:46.914661 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 09:03:46.914692 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:03:46.914693 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:03:46.915983 (+ 1290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:03:46.915984 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:03:46.917155 (+ 1171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:03:46.917158 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:03:46.918165 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:03:46.918166 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:03:46.918229 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:03:46.918234 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 09:03:46.918282 (+ 48us) client_negotiation.cc:240] Negotiation successful
0619 09:03:46.918326 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":438,"mutex_wait_us":8,"thread_start_us":76,"threads_started":1}
I20260619 09:03:46.918550 16264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:03:46.914299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42600 (local address 127.0.98.1:41245)
0619 09:03:46.914750 (+ 451us) server_negotiation.cc:206] Beginning negotiation
0619 09:03:46.914753 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 09:03:46.914763 (+ 10us) server_negotiation.cc:407] Connection header received
0619 09:03:46.914798 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:03:46.914800 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:03:46.914813 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:03:46.914897 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 09:03:46.915001 (+ 104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:03:46.916044 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:03:46.917307 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:03:46.917916 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:03:46.917985 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:03:46.918045 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0619 09:03:46.918433 (+ 388us) server_negotiation.cc:299] Negotiation successful
0619 09:03:46.918476 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":414,"thread_start_us":93,"threads_started":1}
I20260619 09:03:46.918550 16262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:03:46.914258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38536 (local address 127.0.98.2:33867)
0619 09:03:46.914440 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0619 09:03:46.914445 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 09:03:46.914509 (+ 64us) server_negotiation.cc:407] Connection header received
0619 09:03:46.914564 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:03:46.914566 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:03:46.914580 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:03:46.914651 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 09:03:46.914742 (+ 91us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:03:46.915887 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:03:46.917240 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:03:46.917917 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:03:46.918150 (+ 233us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:03:46.918218 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0619 09:03:46.918299 (+ 81us) server_negotiation.cc:299] Negotiation successful
0619 09:03:46.918412 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":79,"threads_started":1}
I20260619 09:03:46.918803 16185 tablet_service.cc:1467] Processing CreateTablet for tablet b79eb15bf8494a988f64747c4f7f072b (DEFAULT_TABLE table=test-table [id=1bb273f426ed4bea88ba80ab42754192]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 09:03:46.919178 16185 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b79eb15bf8494a988f64747c4f7f072b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 09:03:46.919173 15917 tablet_service.cc:1467] Processing CreateTablet for tablet b79eb15bf8494a988f64747c4f7f072b (DEFAULT_TABLE table=test-table [id=1bb273f426ed4bea88ba80ab42754192]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 09:03:46.919231 16051 tablet_service.cc:1467] Processing CreateTablet for tablet b79eb15bf8494a988f64747c4f7f072b (DEFAULT_TABLE table=test-table [id=1bb273f426ed4bea88ba80ab42754192]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 09:03:46.919471 15917 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b79eb15bf8494a988f64747c4f7f072b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 09:03:46.919541 16051 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b79eb15bf8494a988f64747c4f7f072b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 09:03:46.921993 16265 tablet_bootstrap.cc:492] T b79eb15bf8494a988f64747c4f7f072b P c9252303f6c147b99263a5c7666a6134: Bootstrap starting.
I20260619 09:03:46.922577 16265 tablet_bootstrap.cc:654] T b79eb15bf8494a988f64747c4f7f072b P c9252303f6c147b99263a5c7666a6134: Neither blocks nor log segments found. Creating new log.
I20260619 09:03:46.922890 16266 tablet_bootstrap.cc:492] T b79eb15bf8494a988f64747c4f7f072b P 51ec1359afd84648a293aff68cab2a8e: Bootstrap starting.
I20260619 09:03:46.922941 16267 tablet_bootstrap.cc:492] T b79eb15bf8494a988f64747c4f7f072b P ce39db3fe58a4151a4795ecaf0942bc9: Bootstrap starting.
I20260619 09:03:46.923318 16265 log.cc:824] T b79eb15bf8494a988f64747c4f7f072b P c9252303f6c147b99263a5c7666a6134: Log is configured to *not* fsync() on all Append() calls
I20260619 09:03:46.923482 16267 tablet_bootstrap.cc:654] T b79eb15bf8494a988f64747c4f7f072b P ce39db3fe58a4151a4795ecaf0942bc9: Neither blocks nor log segments found. Creating new log.
I20260619 09:03:46.923511 16266 tablet_bootstrap.cc:654] T b79eb15bf8494a988f64747c4f7f072b P 51ec1359afd84648a293aff68cab2a8e: Neither blocks nor log segments found. Creating new log.
I20260619 09:03:46.923913 16266 log.cc:824] T b79eb15bf8494a988f64747c4f7f072b P 51ec1359afd84648a293aff68cab2a8e: Log is configured to *not* fsync() on all Append() calls
I20260619 09:03:46.923911 16267 log.cc:824] T b79eb15bf8494a988f64747c4f7f072b P ce39db3fe58a4151a4795ecaf0942bc9: Log is configured to *not* fsync() on all Append() calls
I20260619 09:03:46.924490 16266 tablet_bootstrap.cc:492] T b79eb15bf8494a988f64747c4f7f072b P 51ec1359afd84648a293aff68cab2a8e: No bootstrap required, opened a new log
I20260619 09:03:46.924502 16267 tablet_bootstrap.cc:492] T b79eb15bf8494a988f64747c4f7f072b P ce39db3fe58a4151a4795ecaf0942bc9: No bootstrap required, opened a new log
I20260619 09:03:46.924540 16266 ts_tablet_manager.cc:1397] T b79eb15bf8494a988f64747c4f7f072b P 51ec1359afd84648a293aff68cab2a8e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 09:03:46.924548 16267 ts_tablet_manager.cc:1397] T b79eb15bf8494a988f64747c4f7f072b P ce39db3fe58a4151a4795ecaf0942bc9: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 09:03:46.925930 16266 raft_consensus.cc:348] T b79eb15bf8494a988f64747c4f7f072b P 51ec1359afd84648a293aff68cab2a8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce39db3fe58a4151a4795ecaf0942bc9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46739 } } peers { permanent_uuid: "51ec1359afd84648a293aff68cab2a8e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33867 } } peers { permanent_uuid: "c9252303f6c147b99263a5c7666a6134" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41245 } }
I20260619 09:03:46.926074 16266 raft_consensus.cc:374] T b79eb15bf8494a988f64747c4f7f072b P 51ec1359afd84648a293aff68cab2a8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 09:03:46.925983 16267 raft_consensus.cc:348] T b79eb15bf8494a988f64747c4f7f072b P ce39db3fe58a4151a4795ecaf0942bc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce39db3fe58a4151a4795ecaf0942bc9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46739 } } peers { permanent_uuid: "51ec1359afd84648a293aff68cab2a8e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33867 } } peers { permanent_uuid: "c9252303f6c147b99263a5c7666a6134" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41245 } }
I20260619 09:03:46.926096 16266 raft_consensus.cc:729] T b79eb15bf8494a988f64747c4f7f072b P 51ec1359afd84648a293aff68cab2a8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51ec1359afd84648a293aff68cab2a8e, State: Initialized, Role: FOLLOWER
I20260619 09:03:46.926108 16267 raft_consensus.cc:374] T b79eb15bf8494a988f64747c4f7f072b P ce39db3fe58a4151a4795ecaf0942bc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 09:03:46.926149 16267 raft_consensus.cc:729] T b79eb15bf8494a988f64747c4f7f072b P ce39db3fe58a4151a4795ecaf0942bc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce39db3fe58a4151a4795ecaf0942bc9, State: Initialized, Role: FOLLOWER
I20260619 09:03:46.926211 16266 consensus_queue.cc:260] T b79eb15bf8494a988f64747c4f7f072b P 51ec1359afd84648a293aff68cab2a8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce39db3fe58a4151a4795ecaf0942bc9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46739 } } peers { permanent_uuid: "51ec1359afd84648a293aff68cab2a8e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33867 } } peers { permanent_uuid: "c9252303f6c147b99263a5c7666a6134" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41245 } }
I20260619 09:03:46.926254 16267 consensus_queue.cc:260] T b79eb15bf8494a988f64747c4f7f072b P ce39db3fe58a4151a4795ecaf0942bc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce39db3fe58a4151a4795ecaf0942bc9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46739 } } peers { permanent_uuid: "51ec1359afd84648a293aff68cab2a8e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33867 } } peers { permanent_uuid: "c9252303f6c147b99263a5c7666a6134" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41245 } }
I20260619 09:03:46.926476 16117 heartbeater.cc:503] Master 127.0.98.62:42887 was elected leader, sending a full tablet report...
I20260619 09:03:46.926502 16267 ts_tablet_manager.cc:1428] T b79eb15bf8494a988f64747c4f7f072b P ce39db3fe58a4151a4795ecaf0942bc9: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 09:03:46.926730 16251 heartbeater.cc:503] Master 127.0.98.62:42887 was elected leader, sending a full tablet report...
I20260619 09:03:46.926908 16265 tablet_bootstrap.cc:492] T b79eb15bf8494a988f64747c4f7f072b P c9252303f6c147b99263a5c7666a6134: No bootstrap required, opened a new log
I20260619 09:03:46.926971 16265 ts_tablet_manager.cc:1397] T b79eb15bf8494a988f64747c4f7f072b P c9252303f6c147b99263a5c7666a6134: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.001s
I20260619 09:03:46.926460 16266 ts_tablet_manager.cc:1428] T b79eb15bf8494a988f64747c4f7f072b P 51ec1359afd84648a293aff68cab2a8e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 09:03:46.928398 16265 raft_consensus.cc:348] T b79eb15bf8494a988f64747c4f7f072b P c9252303f6c147b99263a5c7666a6134 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce39db3fe58a4151a4795ecaf0942bc9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46739 } } peers { permanent_uuid: "51ec1359afd84648a293aff68cab2a8e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33867 } } peers { permanent_uuid: "c9252303f6c147b99263a5c7666a6134" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41245 } }
I20260619 09:03:46.928537 16265 raft_consensus.cc:374] T b79eb15bf8494a988f64747c4f7f072b P c9252303f6c147b99263a5c7666a6134 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 09:03:46.928558 16265 raft_consensus.cc:729] T b79eb15bf8494a988f64747c4f7f072b P c9252303f6c147b99263a5c7666a6134 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9252303f6c147b99263a5c7666a6134, State: Initialized, Role: FOLLOWER
I20260619 09:03:46.928637 16265 consensus_queue.cc:260] T b79eb15bf8494a988f64747c4f7f072b P c9252303f6c147b99263a5c7666a6134 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce39db3fe58a4151a4795ecaf0942bc9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46739 } } peers { permanent_uuid: "51ec1359afd84648a293aff68cab2a8e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33867 } } peers { permanent_uuid: "c9252303f6c147b99263a5c7666a6134" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41245 } }
I20260619 09:03:46.928880 16265 ts_tablet_manager.cc:1428] T b79eb15bf8494a988f64747c4f7f072b P c9252303f6c147b99263a5c7666a6134: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260619 09:03:46.928910 15983 heartbeater.cc:503] Master 127.0.98.62:42887 was elected leader, sending a full tablet report...
I20260619 09:03:46.985215 16273 raft_consensus.cc:482] T b79eb15bf8494a988f64747c4f7f072b P c9252303f6c147b99263a5c7666a6134 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 09:03:46.985351 16273 raft_consensus.cc:504] T b79eb15bf8494a988f64747c4f7f072b P c9252303f6c147b99263a5c7666a6134 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce39db3fe58a4151a4795ecaf0942bc9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46739 } } peers { permanent_uuid: "51ec1359afd84648a293aff68cab2a8e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33867 } } peers { permanent_uuid: "c9252303f6c147b99263a5c7666a6134" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41245 } }
I20260619 09:03:46.985678 16273 leader_election.cc:288] T b79eb15bf8494a988f64747c4f7f072b P c9252303f6c147b99263a5c7666a6134 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ce39db3fe58a4151a4795ecaf0942bc9 (127.0.98.3:46739), 51ec1359afd84648a293aff68cab2a8e (127.0.98.2:33867)
I20260619 09:03:46.988484 16262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:03:46.985990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37331 (local address 127.0.98.2:33867)
0619 09:03:46.986035 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0619 09:03:46.986037 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:03:46.986048 (+ 11us) server_negotiation.cc:407] Connection header received
0619 09:03:46.986076 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:03:46.986077 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:03:46.986090 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:03:46.986144 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 09:03:46.986313 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:03:46.986802 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:03:46.987749 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:03:46.988126 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:03:46.988178 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:03:46.988186 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0619 09:03:46.988356 (+ 170us) server_negotiation.cc:299] Negotiation successful
0619 09:03:46.988397 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260619 09:03:46.988490 16263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:03:46.985960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53671 (local address 127.0.98.3:46739)
0619 09:03:46.986021 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0619 09:03:46.986023 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:03:46.986142 (+ 119us) server_negotiation.cc:407] Connection header received
0619 09:03:46.986177 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:03:46.986178 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:03:46.986189 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:03:46.986253 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 09:03:46.986395 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:03:46.986928 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:03:46.987813 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:03:46.988189 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:03:46.988275 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:03:46.988285 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0619 09:03:46.988384 (+ 99us) server_negotiation.cc:299] Negotiation successful
0619 09:03:46.988423 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260619 09:03:46.988499 16274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:03:46.985849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46739 (local address 127.0.98.1:53671)
0619 09:03:46.986070 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0619 09:03:46.986078 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0619 09:03:46.986137 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:03:46.986254 (+ 117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:03:46.986255 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:03:46.986256 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 09:03:46.986323 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:03:46.986324 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:03:46.987001 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:03:46.987002 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:03:46.987721 (+ 719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:03:46.987723 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:03:46.988265 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:03:46.988265 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:03:46.988318 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:03:46.988323 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 09:03:46.988367 (+ 44us) client_negotiation.cc:240] Negotiation successful
0619 09:03:46.988401 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":205,"mutex_wait_us":27,"thread_start_us":100,"threads_started":1}
I20260619 09:03:46.988498 15985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:03:46.985849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33867 (local address 127.0.98.1:37331)
0619 09:03:46.985905 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0619 09:03:46.985918 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 09:03:46.985980 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:03:46.986158 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:03:46.986159 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:03:46.986161 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 09:03:46.986242 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:03:46.986244 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:03:46.986888 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:03:46.986889 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:03:46.987662 (+ 773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:03:46.987664 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:03:46.988195 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:03:46.988196 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:03:46.988277 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:03:46.988281 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 09:03:46.988334 (+ 53us) client_negotiation.cc:240] Negotiation successful
0619 09:03:46.988375 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260619 09:03:46.988848 16071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b79eb15bf8494a988f64747c4f7f072b" candidate_uuid: "c9252303f6c147b99263a5c7666a6134" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51ec1359afd84648a293aff68cab2a8e" is_pre_election: true
I20260619 09:03:46.988893 16205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b79eb15bf8494a988f64747c4f7f072b" candidate_uuid: "c9252303f6c147b99263a5c7666a6134" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce39db3fe58a4151a4795ecaf0942bc9" is_pre_election: true
I20260619 09:03:46.989027 16205 raft_consensus.cc:2454] T b79eb15bf8494a988f64747c4f7f072b P ce39db3fe58a4151a4795ecaf0942bc9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c9252303f6c147b99263a5c7666a6134 in term 0.
I20260619 09:03:46.989029 16071 raft_consensus.cc:2454] T b79eb15bf8494a988f64747c4f7f072b P 51ec1359afd84648a293aff68cab2a8e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c9252303f6c147b99263a5c7666a6134 in term 0.
I20260619 09:03:46.989255 15870 leader_election.cc:302] T b79eb15bf8494a988f64747c4f7f072b P c9252303f6c147b99263a5c7666a6134 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c9252303f6c147b99263a5c7666a6134, ce39db3fe58a4151a4795ecaf0942bc9; no voters:
I20260619 09:03:46.989395 16273 raft_consensus.cc:2781] T b79eb15bf8494a988f64747c4f7f072b P c9252303f6c147b99263a5c7666a6134 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 09:03:46.989465 16273 raft_consensus.cc:482] T b79eb15bf8494a988f64747c4f7f072b P c9252303f6c147b99263a5c7666a6134 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 09:03:46.989492 16273 raft_consensus.cc:3037] T b79eb15bf8494a988f64747c4f7f072b P c9252303f6c147b99263a5c7666a6134 [term 0 FOLLOWER]: Advancing to term 1
I20260619 09:03:46.990301 16273 raft_consensus.cc:504] T b79eb15bf8494a988f64747c4f7f072b P c9252303f6c147b99263a5c7666a6134 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce39db3fe58a4151a4795ecaf0942bc9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46739 } } peers { permanent_uuid: "51ec1359afd84648a293aff68cab2a8e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33867 } } peers { permanent_uuid: "c9252303f6c147b99263a5c7666a6134" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41245 } }
I20260619 09:03:46.990432 16273 leader_election.cc:288] T b79eb15bf8494a988f64747c4f7f072b P c9252303f6c147b99263a5c7666a6134 [CANDIDATE]: Term 1 election: Requested vote from peers ce39db3fe58a4151a4795ecaf0942bc9 (127.0.98.3:46739), 51ec1359afd84648a293aff68cab2a8e (127.0.98.2:33867)
I20260619 09:03:46.990605 16205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b79eb15bf8494a988f64747c4f7f072b" candidate_uuid: "c9252303f6c147b99263a5c7666a6134" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce39db3fe58a4151a4795ecaf0942bc9"
I20260619 09:03:46.990635 16071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b79eb15bf8494a988f64747c4f7f072b" candidate_uuid: "c9252303f6c147b99263a5c7666a6134" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51ec1359afd84648a293aff68cab2a8e"
I20260619 09:03:46.990693 16205 raft_consensus.cc:3037] T b79eb15bf8494a988f64747c4f7f072b P ce39db3fe58a4151a4795ecaf0942bc9 [term 0 FOLLOWER]: Advancing to term 1
I20260619 09:03:46.990705 16071 raft_consensus.cc:3037] T b79eb15bf8494a988f64747c4f7f072b P 51ec1359afd84648a293aff68cab2a8e [term 0 FOLLOWER]: Advancing to term 1
I20260619 09:03:46.991590 16205 raft_consensus.cc:2454] T b79eb15bf8494a988f64747c4f7f072b P ce39db3fe58a4151a4795ecaf0942bc9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9252303f6c147b99263a5c7666a6134 in term 1.
I20260619 09:03:46.991590 16071 raft_consensus.cc:2454] T b79eb15bf8494a988f64747c4f7f072b P 51ec1359afd84648a293aff68cab2a8e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9252303f6c147b99263a5c7666a6134 in term 1.
I20260619 09:03:46.991755 15870 leader_election.cc:302] T b79eb15bf8494a988f64747c4f7f072b P c9252303f6c147b99263a5c7666a6134 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c9252303f6c147b99263a5c7666a6134, ce39db3fe58a4151a4795ecaf0942bc9; no voters:
I20260619 09:03:46.991839 16273 raft_consensus.cc:2781] T b79eb15bf8494a988f64747c4f7f072b P c9252303f6c147b99263a5c7666a6134 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 09:03:46.991978 16273 raft_consensus.cc:686] T b79eb15bf8494a988f64747c4f7f072b P c9252303f6c147b99263a5c7666a6134 [term 1 LEADER]: Becoming Leader. State: Replica: c9252303f6c147b99263a5c7666a6134, State: Running, Role: LEADER
I20260619 09:03:46.992087 16273 consensus_queue.cc:237] T b79eb15bf8494a988f64747c4f7f072b P c9252303f6c147b99263a5c7666a6134 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce39db3fe58a4151a4795ecaf0942bc9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46739 } } peers { permanent_uuid: "51ec1359afd84648a293aff68cab2a8e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33867 } } peers { permanent_uuid: "c9252303f6c147b99263a5c7666a6134" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41245 } }
I20260619 09:03:46.992826 15791 catalog_manager.cc:5495] T b79eb15bf8494a988f64747c4f7f072b P c9252303f6c147b99263a5c7666a6134 reported cstate change: term changed from 0 to 1, leader changed from <none> to c9252303f6c147b99263a5c7666a6134 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "c9252303f6c147b99263a5c7666a6134" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce39db3fe58a4151a4795ecaf0942bc9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46739 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "51ec1359afd84648a293aff68cab2a8e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33867 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c9252303f6c147b99263a5c7666a6134" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41245 } health_report { overall_health: HEALTHY } } }
W20260619 09:03:46.999970 16118 tablet.cc:2369] T b79eb15bf8494a988f64747c4f7f072b P 51ec1359afd84648a293aff68cab2a8e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 09:03:47.012260 16264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:03:47.010168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42606 (local address 127.0.98.1:41245)
0619 09:03:47.010221 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0619 09:03:47.010223 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:03:47.010234 (+ 11us) server_negotiation.cc:407] Connection header received
0619 09:03:47.010265 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:03:47.010265 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:03:47.010283 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:03:47.010336 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 09:03:47.010451 (+ 115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:03:47.010965 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:03:47.011662 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:03:47.011835 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:03:47.011877 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:03:47.011998 (+ 121us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 09:03:47.012088 (+ 90us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 09:03:47.012144 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0619 09:03:47.012156 (+ 12us) server_negotiation.cc:299] Negotiation successful
0619 09:03:47.012172 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260619 09:03:47.013957 16071 raft_consensus.cc:1261] T b79eb15bf8494a988f64747c4f7f072b P 51ec1359afd84648a293aff68cab2a8e [term 1 FOLLOWER]: Refusing update from remote peer c9252303f6c147b99263a5c7666a6134: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 09:03:47.013967 16205 raft_consensus.cc:1261] T b79eb15bf8494a988f64747c4f7f072b P ce39db3fe58a4151a4795ecaf0942bc9 [term 1 FOLLOWER]: Refusing update from remote peer c9252303f6c147b99263a5c7666a6134: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 09:03:47.014269 16273 consensus_queue.cc:1035] T b79eb15bf8494a988f64747c4f7f072b P c9252303f6c147b99263a5c7666a6134 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51ec1359afd84648a293aff68cab2a8e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 09:03:47.014401 16275 consensus_queue.cc:1035] T b79eb15bf8494a988f64747c4f7f072b P c9252303f6c147b99263a5c7666a6134 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce39db3fe58a4151a4795ecaf0942bc9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46739 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 09:03:47.016410 16278 mvcc.cc:205] Tried to move back new op lower bound from 7298497851446996992 to 7298497851360501760. Current Snapshot: MvccSnapshot[applied={T|T < 7298497851446996992}]
I20260619 09:03:47.016991 16281 mvcc.cc:205] Tried to move back new op lower bound from 7298497851446996992 to 7298497851360501760. Current Snapshot: MvccSnapshot[applied={T|T < 7298497851446996992}]
I20260619 09:03:47.018927 15792 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49318:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 09:03:47.019003 15792 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49318:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 09:03:47.019631 15792 catalog_manager.cc:5782] T 00000000000000000000000000000000 P df0d2dbbe48b4c98a080b5e59d3c993d: Sending DeleteTablet for 3 replicas of tablet b79eb15bf8494a988f64747c4f7f072b
I20260619 09:03:47.019909 15917 tablet_service.cc:1514] Processing DeleteTablet for tablet b79eb15bf8494a988f64747c4f7f072b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 09:03:47 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42600
I20260619 09:03:47.019923 16051 tablet_service.cc:1514] Processing DeleteTablet for tablet b79eb15bf8494a988f64747c4f7f072b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 09:03:47 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38536
I20260619 09:03:47.019999 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ez5NB/build/release/bin/kudu with pid 15854
I20260619 09:03:47.020125 16185 tablet_service.cc:1514] Processing DeleteTablet for tablet b79eb15bf8494a988f64747c4f7f072b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 09:03:47 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55060
I20260619 09:03:47.020701 16288 tablet_replica.cc:330] T b79eb15bf8494a988f64747c4f7f072b P 51ec1359afd84648a293aff68cab2a8e: stopping tablet replica
I20260619 09:03:47.020794 16288 raft_consensus.cc:2229] T b79eb15bf8494a988f64747c4f7f072b P 51ec1359afd84648a293aff68cab2a8e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 09:03:47.020887 16288 raft_consensus.cc:2258] T b79eb15bf8494a988f64747c4f7f072b P 51ec1359afd84648a293aff68cab2a8e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 09:03:47.021721 16288 ts_tablet_manager.cc:1905] T b79eb15bf8494a988f64747c4f7f072b P 51ec1359afd84648a293aff68cab2a8e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 09:03:47.022917 16290 tablet_replica.cc:330] T b79eb15bf8494a988f64747c4f7f072b P ce39db3fe58a4151a4795ecaf0942bc9: stopping tablet replica
I20260619 09:03:47.022997 16290 raft_consensus.cc:2229] T b79eb15bf8494a988f64747c4f7f072b P ce39db3fe58a4151a4795ecaf0942bc9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 09:03:47.023072 16288 ts_tablet_manager.cc:1918] T b79eb15bf8494a988f64747c4f7f072b P 51ec1359afd84648a293aff68cab2a8e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 09:03:47.023092 16290 raft_consensus.cc:2258] T b79eb15bf8494a988f64747c4f7f072b P ce39db3fe58a4151a4795ecaf0942bc9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 09:03:47.023136 16288 log.cc:1196] T b79eb15bf8494a988f64747c4f7f072b P 51ec1359afd84648a293aff68cab2a8e: Deleting WAL directory at /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859658134510-392-0/minicluster-data/ts-1/wal/wals/b79eb15bf8494a988f64747c4f7f072b
I20260619 09:03:47.023391 16288 ts_tablet_manager.cc:1939] T b79eb15bf8494a988f64747c4f7f072b P 51ec1359afd84648a293aff68cab2a8e: Deleting consensus metadata
I20260619 09:03:47.023495 16290 ts_tablet_manager.cc:1905] T b79eb15bf8494a988f64747c4f7f072b P ce39db3fe58a4151a4795ecaf0942bc9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 09:03:47.023778 15779 catalog_manager.cc:4841] TS 51ec1359afd84648a293aff68cab2a8e (127.0.98.2:33867): tablet b79eb15bf8494a988f64747c4f7f072b (table test-table [id=1bb273f426ed4bea88ba80ab42754192]) successfully deleted
I20260619 09:03:47.024813 16290 ts_tablet_manager.cc:1918] T b79eb15bf8494a988f64747c4f7f072b P ce39db3fe58a4151a4795ecaf0942bc9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 09:03:47.024873 16290 log.cc:1196] T b79eb15bf8494a988f64747c4f7f072b P ce39db3fe58a4151a4795ecaf0942bc9: Deleting WAL directory at /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859658134510-392-0/minicluster-data/ts-2/wal/wals/b79eb15bf8494a988f64747c4f7f072b
I20260619 09:03:47.025121 16290 ts_tablet_manager.cc:1939] T b79eb15bf8494a988f64747c4f7f072b P ce39db3fe58a4151a4795ecaf0942bc9: Deleting consensus metadata
W20260619 09:03:47.025341 15777 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:41245 (error 108)
I20260619 09:03:47.025615 15776 catalog_manager.cc:4841] TS ce39db3fe58a4151a4795ecaf0942bc9 (127.0.98.3:46739): tablet b79eb15bf8494a988f64747c4f7f072b (table test-table [id=1bb273f426ed4bea88ba80ab42754192]) successfully deleted
I20260619 09:03:47.025768 16261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:03:47.025651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41245 (local address 127.0.0.1:42614)
0619 09:03:47.025703 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0619 09:03:47.025740 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260619 09:03:47.025792 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ez5NB/build/release/bin/kudu with pid 15988
W20260619 09:03:47.025918 15777 catalog_manager.cc:4584] TS c9252303f6c147b99263a5c7666a6134 (127.0.98.1:41245): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b79eb15bf8494a988f64747c4f7f072b: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41245: connect: Connection refused (error 111)
I20260619 09:03:47.029549 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ez5NB/build/release/bin/kudu with pid 16122
I20260619 09:03:47.033542 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ez5NB/build/release/bin/kudu with pid 15760
I20260619 09:04:07.037911 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ez5NB/build/release/bin/kudu
/tmp/dist-test-task4ez5NB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859658134510-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859658134510-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859658134510-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859658134510-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42887
--webserver_interface=127.0.98.62
--webserver_port=42285
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859658134510-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42887
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859658134510-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859658134510-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859658134510-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 09:04:07.117540 16292 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 09:04:07.117733 16292 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 09:04:07.117753 16292 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 09:04:07.118866 16292 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260619 09:04:07.118908 16292 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260619 09:04:07.118921 16292 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260619 09:04:07.118932 16292 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 09:04:07.118943 16292 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 09:04:07.118954 16292 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 09:04:07.118965 16292 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 09:04:07.118976 16292 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 09:04:07.120496 16292 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859658134510-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859658134510-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42887
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859658134510-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42887
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859658134510-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=42285
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16292
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859658134510-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 09:04:07.120698 16292 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 09:04:07.120874 16292 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 09:04:07.128934 16301 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:04:07.128957 16299 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:04:07.129067 16292 server_base.cc:1034] running on GCE node
W20260619 09:04:07.128985 16298 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:04:07.129427 16292 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 09:04:07.129706 16292 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 09:04:07.130903 16292 hybrid_clock.cc:648] HybridClock initialized: now 1781859847130887 us; error 29 us; skew 500 ppm
Jun 19 09:04:07 dist-test-slave-mtdr krb5kdc[15747](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859847, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 09:04:07.136294 16292 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 09:04:07.136806 16292 webserver.cc:460] Webserver started at http://127.0.98.62:42285/ using document root <none> and password file <none>
I20260619 09:04:07.137080 16292 fs_manager.cc:359] Metadata directory not provided
I20260619 09:04:07.137152 16292 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 09:04:07.138558 16292 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260619 09:04:07.139353 16308 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 09:04:07.139528 16292 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260619 09:04:07.139614 16292 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859658134510-392-0/minicluster-data/master-0/data,/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859658134510-392-0/minicluster-data/master-0/wal
uuid: "df0d2dbbe48b4c98a080b5e59d3c993d"
format_stamp: "Formatted at 2026-06-19 09:03:46 on dist-test-slave-mtdr"
server_key: "2de344460d2ca414a0dcf14bc1f4db7b"
server_key_iv: "8dfc905c00a66b785545414fb96c3160"
server_key_version: "encryptionkey@0"
I20260619 09:04:07.140033 16292 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859658134510-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859658134510-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859658134510-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 09:04:07.168169 16292 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 09:04:07.170420 16292 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 09:04:07.170567 16292 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 09:04:07.174621 16292 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42887
I20260619 09:04:07.174664 16360 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42887 every 8 connection(s)
I20260619 09:04:07.175081 16292 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859658134510-392-0/minicluster-data/master-0/data/info.pb
I20260619 09:04:07.176494 16361 sys_catalog.cc:263] Verifying existing consensus state
I20260619 09:04:07.177079 16361 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df0d2dbbe48b4c98a080b5e59d3c993d: Bootstrap starting.
I20260619 09:04:07.182919 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ez5NB/build/release/bin/kudu as pid 16292
I20260619 09:04:07.183244 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ez5NB/build/release/bin/kudu
/tmp/dist-test-task4ez5NB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859658134510-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859658134510-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859658134510-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859658134510-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:41245
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:42887
--webserver_port=34623
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859658134510-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859658134510-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859658134510-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859658134510-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 09:04:07.185382 16361 log.cc:824] T 00000000000000000000000000000000 P df0d2dbbe48b4c98a080b5e59d3c993d: Log is configured to *not* fsync() on all Append() calls
I20260619 09:04:07.187269 16361 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df0d2dbbe48b4c98a080b5e59d3c993d: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 09:04:07.187505 16361 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df0d2dbbe48b4c98a080b5e59d3c993d: Bootstrap complete.
I20260619 09:04:07.189525 16361 raft_consensus.cc:348] T 00000000000000000000000000000000 P df0d2dbbe48b4c98a080b5e59d3c993d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df0d2dbbe48b4c98a080b5e59d3c993d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42887 } }
I20260619 09:04:07.189739 16361 raft_consensus.cc:729] T 00000000000000000000000000000000 P df0d2dbbe48b4c98a080b5e59d3c993d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: df0d2dbbe48b4c98a080b5e59d3c993d, State: Initialized, Role: FOLLOWER
I20260619 09:04:07.189880 16361 consensus_queue.cc:260] T 00000000000000000000000000000000 P df0d2dbbe48b4c98a080b5e59d3c993d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df0d2dbbe48b4c98a080b5e59d3c993d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42887 } }
I20260619 09:04:07.189939 16361 raft_consensus.cc:388] T 00000000000000000000000000000000 P df0d2dbbe48b4c98a080b5e59d3c993d [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 09:04:07.189970 16361 raft_consensus.cc:482] T 00000000000000000000000000000000 P df0d2dbbe48b4c98a080b5e59d3c993d [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 09:04:07.190016 16361 raft_consensus.cc:3037] T 00000000000000000000000000000000 P df0d2dbbe48b4c98a080b5e59d3c993d [term 1 FOLLOWER]: Advancing to term 2
I20260619 09:04:07.190999 16361 raft_consensus.cc:504] T 00000000000000000000000000000000 P df0d2dbbe48b4c98a080b5e59d3c993d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df0d2dbbe48b4c98a080b5e59d3c993d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42887 } }
I20260619 09:04:07.191085 16361 leader_election.cc:302] T 00000000000000000000000000000000 P df0d2dbbe48b4c98a080b5e59d3c993d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: df0d2dbbe48b4c98a080b5e59d3c993d; no voters:
I20260619 09:04:07.191231 16361 leader_election.cc:288] T 00000000000000000000000000000000 P df0d2dbbe48b4c98a080b5e59d3c993d [CANDIDATE]: Term 2 election: Requested vote from peers
I20260619 09:04:07.191275 16365 raft_consensus.cc:2781] T 00000000000000000000000000000000 P df0d2dbbe48b4c98a080b5e59d3c993d [term 2 FOLLOWER]: Leader election won for term 2
I20260619 09:04:07.191457 16365 raft_consensus.cc:686] T 00000000000000000000000000000000 P df0d2dbbe48b4c98a080b5e59d3c993d [term 2 LEADER]: Becoming Leader. State: Replica: df0d2dbbe48b4c98a080b5e59d3c993d, State: Running, Role: LEADER
I20260619 09:04:07.191478 16361 sys_catalog.cc:564] T 00000000000000000000000000000000 P df0d2dbbe48b4c98a080b5e59d3c993d [sys.catalog]: configured and running, proceeding with master startup.
I20260619 09:04:07.191596 16365 consensus_queue.cc:237] T 00000000000000000000000000000000 P df0d2dbbe48b4c98a080b5e59d3c993d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df0d2dbbe48b4c98a080b5e59d3c993d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42887 } }
I20260619 09:04:07.191999 16367 sys_catalog.cc:455] T 00000000000000000000000000000000 P df0d2dbbe48b4c98a080b5e59d3c993d [sys.catalog]: SysCatalogTable state changed. Reason: New leader df0d2dbbe48b4c98a080b5e59d3c993d. Latest consensus state: current_term: 2 leader_uuid: "df0d2dbbe48b4c98a080b5e59d3c993d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df0d2dbbe48b4c98a080b5e59d3c993d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42887 } } }
I20260619 09:04:07.192009 16366 sys_catalog.cc:455] T 00000000000000000000000000000000 P df0d2dbbe48b4c98a080b5e59d3c993d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "df0d2dbbe48b4c98a080b5e59d3c993d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df0d2dbbe48b4c98a080b5e59d3c993d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42887 } } }
I20260619 09:04:07.192092 16367 sys_catalog.cc:458] T 00000000000000000000000000000000 P df0d2dbbe48b4c98a080b5e59d3c993d [sys.catalog]: This master's current role is: LEADER
I20260619 09:04:07.192106 16366 sys_catalog.cc:458] T 00000000000000000000000000000000 P df0d2dbbe48b4c98a080b5e59d3c993d [sys.catalog]: This master's current role is: LEADER
I20260619 09:04:07.192286 16372 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 09:04:07.193176 16372 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 09:04:07.193293 16372 catalog_manager.cc:1259] Loaded cluster ID: fb77a7cd21ee4af78f446a5fba7eb31e
I20260619 09:04:07.193333 16372 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 09:04:07.194589 16372 catalog_manager.cc:1504] Loading token signing keys...
I20260619 09:04:07.194775 16372 catalog_manager.cc:5879] T 00000000000000000000000000000000 P df0d2dbbe48b4c98a080b5e59d3c993d: Loaded TSK: 0
I20260619 09:04:07.206429 16372 catalog_manager.cc:5868] T 00000000000000000000000000000000 P df0d2dbbe48b4c98a080b5e59d3c993d: Generated new TSK 1
I20260619 09:04:07.206518 16372 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 09:04:07.263763 16363 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 09:04:07.263962 16363 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 09:04:07.263993 16363 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 09:04:07.265117 16363 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 09:04:07.265158 16363 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 09:04:07.265193 16363 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 09:04:07.266592 16363 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859658134510-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859658134510-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859658134510-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:41245
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859658134510-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=34623
--tserver_master_addrs=127.0.98.62:42887
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16363
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859658134510-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 09:04:07.266830 16363 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 09:04:07.267050 16363 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 09:04:07.271595 16389 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:04:07.271672 16388 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:04:07.271894 16391 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:04:07.272099 16363 server_base.cc:1034] running on GCE node
I20260619 09:04:07.272282 16363 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 09:04:07.272526 16363 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 09:04:07.273664 16363 hybrid_clock.cc:648] HybridClock initialized: now 1781859847273651 us; error 30 us; skew 500 ppm
Jun 19 09:04:07 dist-test-slave-mtdr krb5kdc[15747](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859847, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 09:04:07.276086 16363 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 09:04:07.276578 16363 webserver.cc:460] Webserver started at http://127.0.98.1:34623/ using document root <none> and password file <none>
I20260619 09:04:07.276849 16363 fs_manager.cc:359] Metadata directory not provided
I20260619 09:04:07.276932 16363 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 09:04:07.278604 16363 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 09:04:07.279413 16398 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 09:04:07.279606 16363 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260619 09:04:07.279692 16363 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859658134510-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859658134510-392-0/minicluster-data/ts-0/wal
uuid: "c9252303f6c147b99263a5c7666a6134"
format_stamp: "Formatted at 2026-06-19 09:03:46 on dist-test-slave-mtdr"
server_key: "4c23a7631a8101e8f8f522d0837b3dc8"
server_key_iv: "ff9c47905da211813b68b95dda196455"
server_key_version: "encryptionkey@0"
I20260619 09:04:07.280030 16363 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859658134510-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859658134510-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859658134510-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 09:04:07.296443 16363 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 09:04:07.298440 16363 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 09:04:07.298552 16363 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 09:04:07.298769 16363 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 09:04:07.299211 16405 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260619 09:04:07.301051 16363 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260619 09:04:07.301101 16363 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260619 09:04:07.301136 16363 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260619 09:04:07.301728 16363 ts_tablet_manager.cc:610] Registered 1 tablets
I20260619 09:04:07.301770 16363 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260619 09:04:07.301864 16405 tablet_bootstrap.cc:492] T b79eb15bf8494a988f64747c4f7f072b P c9252303f6c147b99263a5c7666a6134: Bootstrap starting.
I20260619 09:04:07.308262 16363 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41245
I20260619 09:04:07.308395 16512 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41245 every 8 connection(s)
I20260619 09:04:07.308696 16363 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859658134510-392-0/minicluster-data/ts-0/data/info.pb
I20260619 09:04:07.312213 16405 log.cc:824] T b79eb15bf8494a988f64747c4f7f072b P c9252303f6c147b99263a5c7666a6134: Log is configured to *not* fsync() on all Append() calls
I20260619 09:04:07.314877 16405 tablet_bootstrap.cc:492] T b79eb15bf8494a988f64747c4f7f072b P c9252303f6c147b99263a5c7666a6134: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 09:04:07.315120 16405 tablet_bootstrap.cc:492] T b79eb15bf8494a988f64747c4f7f072b P c9252303f6c147b99263a5c7666a6134: Bootstrap complete.
I20260619 09:04:07.315939 16405 ts_tablet_manager.cc:1397] T b79eb15bf8494a988f64747c4f7f072b P c9252303f6c147b99263a5c7666a6134: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.008s
I20260619 09:04:07.317282 16405 raft_consensus.cc:348] T b79eb15bf8494a988f64747c4f7f072b P c9252303f6c147b99263a5c7666a6134 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce39db3fe58a4151a4795ecaf0942bc9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46739 } } peers { permanent_uuid: "51ec1359afd84648a293aff68cab2a8e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33867 } } peers { permanent_uuid: "c9252303f6c147b99263a5c7666a6134" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41245 } }
I20260619 09:04:07.317588 16405 raft_consensus.cc:729] T b79eb15bf8494a988f64747c4f7f072b P c9252303f6c147b99263a5c7666a6134 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9252303f6c147b99263a5c7666a6134, State: Initialized, Role: FOLLOWER
I20260619 09:04:07.317732 16405 consensus_queue.cc:260] T b79eb15bf8494a988f64747c4f7f072b P c9252303f6c147b99263a5c7666a6134 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce39db3fe58a4151a4795ecaf0942bc9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46739 } } peers { permanent_uuid: "51ec1359afd84648a293aff68cab2a8e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33867 } } peers { permanent_uuid: "c9252303f6c147b99263a5c7666a6134" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41245 } }
I20260619 09:04:07.317987 16405 ts_tablet_manager.cc:1428] T b79eb15bf8494a988f64747c4f7f072b P c9252303f6c147b99263a5c7666a6134: Time spent starting tablet: real 0.002s user 0.003s sys 0.001s
I20260619 09:04:07.318385 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ez5NB/build/release/bin/kudu as pid 16363
I20260619 09:04:07.318665 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ez5NB/build/release/bin/kudu
/tmp/dist-test-task4ez5NB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859658134510-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859658134510-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859658134510-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859658134510-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:33867
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:42887
--webserver_port=36909
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859658134510-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859658134510-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859658134510-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859658134510-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 09:04:07 dist-test-slave-mtdr krb5kdc[15747](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859847, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 09:04:07.323746 16515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:04:07.309351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42887 (local address 127.0.98.1:47681)
0619 09:04:07.312493 (+ 3142us) negotiation.cc:107] Waiting for socket to connect
0619 09:04:07.312532 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 09:04:07.313168 (+ 636us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:04:07.313818 (+ 650us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:04:07.313822 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:04:07.314168 (+ 346us) client_negotiation.cc:189] Negotiated authn=SASL
0619 09:04:07.314475 (+ 307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:04:07.314482 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:04:07.316182 (+ 1700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:04:07.316183 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:04:07.317443 (+ 1260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:04:07.317446 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:04:07.318308 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:04:07.318309 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:04:07.318511 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:04:07.319202 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 09:04:07.319212 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 09:04:07.320621 (+ 1409us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 09:04:07.322281 (+ 1660us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:04:07.322287 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:04:07.322289 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:04:07.322490 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:04:07.322709 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:04:07.322709 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:04:07.322710 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:04:07.322831 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:04:07.323157 (+ 326us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 09:04:07.323163 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 09:04:07.323298 (+ 135us) client_negotiation.cc:769] Sending connection context
0619 09:04:07.323376 (+ 78us) client_negotiation.cc:240] Negotiation successful
0619 09:04:07.323546 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3100,"thread_start_us":92,"threads_started":1}
I20260619 09:04:07.323746 16516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:04:07.309431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47681 (local address 127.0.98.62:42887)
0619 09:04:07.312802 (+ 3371us) server_negotiation.cc:206] Beginning negotiation
0619 09:04:07.312812 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 09:04:07.312861 (+ 49us) server_negotiation.cc:407] Connection header received
0619 09:04:07.313553 (+ 692us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:04:07.313560 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:04:07.313622 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:04:07.313767 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:04:07.314609 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:04:07.316062 (+ 1453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:04:07.317546 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:04:07.318215 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:04:07.318320 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:04:07.320721 (+ 2401us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:04:07.320726 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:04:07.320726 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:04:07.320738 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:04:07.322204 (+ 1466us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:04:07.322563 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:04:07.322566 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:04:07.322566 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:04:07.322653 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:04:07.322882 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:04:07.322883 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:04:07.322884 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:04:07.323097 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:04:07.323168 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0619 09:04:07.323450 (+ 282us) server_negotiation.cc:299] Negotiation successful
0619 09:04:07.323566 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3306,"thread_start_us":77,"threads_started":1}
I20260619 09:04:07.324384 16513 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42887
I20260619 09:04:07.324473 16513 heartbeater.cc:464] Registering TS with master...
I20260619 09:04:07.324630 16513 heartbeater.cc:511] Master 127.0.98.62:42887 requested a full tablet report, sending...
I20260619 09:04:07.325229 16325 ts_manager.cc:194] Registered new tserver with Master: c9252303f6c147b99263a5c7666a6134 (127.0.98.1:41245)
I20260619 09:04:07.326325 16325 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:47681
I20260619 09:04:07.327171 16513 heartbeater.cc:503] Master 127.0.98.62:42887 was elected leader, sending a full tablet report...
I20260619 09:04:07.331714 16520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:04:07.325765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41245 (local address 127.0.0.1:36336)
0619 09:04:07.328135 (+ 2370us) negotiation.cc:107] Waiting for socket to connect
0619 09:04:07.328173 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 09:04:07.328244 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:04:07.328536 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:04:07.328539 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:04:07.328541 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 09:04:07.328592 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:04:07.328593 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:04:07.329832 (+ 1239us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:04:07.329833 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:04:07.330594 (+ 761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:04:07.330596 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:04:07.331491 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:04:07.331492 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:04:07.331560 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:04:07.331570 (+ 10us) client_negotiation.cc:769] Sending connection context
0619 09:04:07.331626 (+ 56us) client_negotiation.cc:240] Negotiation successful
0619 09:04:07.331663 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2341,"thread_start_us":82,"threads_started":1}
I20260619 09:04:07.331758 16521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:04:07.325992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36336 (local address 127.0.98.1:41245)
0619 09:04:07.328374 (+ 2382us) server_negotiation.cc:206] Beginning negotiation
0619 09:04:07.328378 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 09:04:07.328387 (+ 9us) server_negotiation.cc:407] Connection header received
0619 09:04:07.328419 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:04:07.328421 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:04:07.328436 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:04:07.328514 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 09:04:07.328683 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:04:07.329765 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:04:07.330669 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:04:07.331389 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:04:07.331459 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:04:07.331512 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0619 09:04:07.331653 (+ 141us) server_negotiation.cc:299] Negotiation successful
0619 09:04:07.331688 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2351,"thread_start_us":63,"threads_started":1}
I20260619 09:04:07.332093 16429 tablet_service.cc:1514] Processing DeleteTablet for tablet b79eb15bf8494a988f64747c4f7f072b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 09:03:47 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36336
I20260619 09:04:07.332604 16522 tablet_replica.cc:330] stopping tablet replica
I20260619 09:04:07.332685 16522 raft_consensus.cc:2229] T b79eb15bf8494a988f64747c4f7f072b P c9252303f6c147b99263a5c7666a6134 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 09:04:07.332760 16522 raft_consensus.cc:2258] T b79eb15bf8494a988f64747c4f7f072b P c9252303f6c147b99263a5c7666a6134 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 09:04:07.333036 16522 ts_tablet_manager.cc:1905] T b79eb15bf8494a988f64747c4f7f072b P c9252303f6c147b99263a5c7666a6134: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 09:04:07.334985 16522 ts_tablet_manager.cc:1918] T b79eb15bf8494a988f64747c4f7f072b P c9252303f6c147b99263a5c7666a6134: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 09:04:07.335050 16522 log.cc:1196] T b79eb15bf8494a988f64747c4f7f072b P c9252303f6c147b99263a5c7666a6134: Deleting WAL directory at /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859658134510-392-0/minicluster-data/ts-0/wal/wals/b79eb15bf8494a988f64747c4f7f072b
I20260619 09:04:07.335278 16522 ts_tablet_manager.cc:1939] T b79eb15bf8494a988f64747c4f7f072b P c9252303f6c147b99263a5c7666a6134: Deleting consensus metadata
I20260619 09:04:07.335728 16310 catalog_manager.cc:4841] TS c9252303f6c147b99263a5c7666a6134 (127.0.98.1:41245): tablet b79eb15bf8494a988f64747c4f7f072b (table test-table [id=1bb273f426ed4bea88ba80ab42754192]) successfully deleted
W20260619 09:04:07.399503 16519 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 09:04:07.399703 16519 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 09:04:07.399734 16519 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 09:04:07.400876 16519 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 09:04:07.400919 16519 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 09:04:07.400954 16519 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 09:04:07.402387 16519 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859658134510-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859658134510-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859658134510-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:33867
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859658134510-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=36909
--tserver_master_addrs=127.0.98.62:42887
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16519
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859658134510-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 09:04:07.402617 16519 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 09:04:07.402843 16519 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 09:04:07.407548 16527 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:04:07.407558 16528 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:04:07.407558 16530 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:04:07.407727 16519 server_base.cc:1034] running on GCE node
I20260619 09:04:07.407930 16519 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 09:04:07.408138 16519 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 09:04:07.409268 16519 hybrid_clock.cc:648] HybridClock initialized: now 1781859847409258 us; error 25 us; skew 500 ppm
Jun 19 09:04:07 dist-test-slave-mtdr krb5kdc[15747](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859847, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 09:04:07.411680 16519 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 09:04:07.412086 16519 webserver.cc:460] Webserver started at http://127.0.98.2:36909/ using document root <none> and password file <none>
I20260619 09:04:07.412318 16519 fs_manager.cc:359] Metadata directory not provided
I20260619 09:04:07.412376 16519 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 09:04:07.413700 16519 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260619 09:04:07.414345 16537 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 09:04:07.414511 16519 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260619 09:04:07.414592 16519 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859658134510-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859658134510-392-0/minicluster-data/ts-1/wal
uuid: "51ec1359afd84648a293aff68cab2a8e"
format_stamp: "Formatted at 2026-06-19 09:03:46 on dist-test-slave-mtdr"
server_key: "3c6eeecd77c8caf9312c2c16884bba17"
server_key_iv: "5bce3236820d4114c2c8a452b1ee9625"
server_key_version: "encryptionkey@0"
I20260619 09:04:07.414948 16519 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859658134510-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859658134510-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859658134510-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 09:04:07.443441 16519 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 09:04:07.445605 16519 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 09:04:07.445724 16519 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 09:04:07.445935 16519 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 09:04:07.446213 16519 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 09:04:07.446246 16519 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 09:04:07.446269 16519 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 09:04:07.446280 16519 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 09:04:07.451566 16519 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33867
I20260619 09:04:07.451617 16650 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33867 every 8 connection(s)
I20260619 09:04:07.451922 16519 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859658134510-392-0/minicluster-data/ts-1/data/info.pb
I20260619 09:04:07.453872 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ez5NB/build/release/bin/kudu as pid 16519
I20260619 09:04:07.454109 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ez5NB/build/release/bin/kudu
/tmp/dist-test-task4ez5NB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859658134510-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859658134510-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859658134510-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859658134510-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:46739
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:42887
--webserver_port=38789
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859658134510-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859658134510-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859658134510-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859658134510-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 09:04:07 dist-test-slave-mtdr krb5kdc[15747](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859847, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 09:04:07.461112 16516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:04:07.452613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53217 (local address 127.0.98.62:42887)
0619 09:04:07.452704 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0619 09:04:07.452706 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:04:07.452851 (+ 145us) server_negotiation.cc:407] Connection header received
0619 09:04:07.453354 (+ 503us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:04:07.453355 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:04:07.453393 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:04:07.453441 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:04:07.454472 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:04:07.455026 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:04:07.456025 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:04:07.456233 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:04:07.456289 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:04:07.458585 (+ 2296us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:04:07.458587 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:04:07.458587 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:04:07.458595 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:04:07.459897 (+ 1302us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:04:07.460257 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:04:07.460258 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:04:07.460259 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:04:07.460311 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:04:07.460517 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:04:07.460518 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:04:07.460518 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:04:07.460624 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:04:07.460680 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0619 09:04:07.460931 (+ 251us) server_negotiation.cc:299] Negotiation successful
0619 09:04:07.461020 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 09:04:07.461287 16653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:04:07.452568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42887 (local address 127.0.98.2:53217)
0619 09:04:07.452736 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0619 09:04:07.452774 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 09:04:07.453270 (+ 496us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:04:07.453512 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:04:07.453515 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:04:07.454123 (+ 608us) client_negotiation.cc:189] Negotiated authn=SASL
0619 09:04:07.454381 (+ 258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:04:07.454386 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:04:07.455113 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:04:07.455115 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:04:07.455942 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:04:07.455945 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:04:07.456301 (+ 356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:04:07.456302 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:04:07.456433 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:04:07.456924 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 09:04:07.456933 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0619 09:04:07.458484 (+ 1551us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 09:04:07.459966 (+ 1482us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:04:07.459973 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:04:07.459975 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:04:07.460190 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:04:07.460364 (+ 174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:04:07.460365 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:04:07.460365 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:04:07.460463 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:04:07.460676 (+ 213us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 09:04:07.460682 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 09:04:07.460815 (+ 133us) client_negotiation.cc:769] Sending connection context
0619 09:04:07.460904 (+ 89us) client_negotiation.cc:240] Negotiation successful
0619 09:04:07.461058 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":80,"threads_started":1}
I20260619 09:04:07.461679 16651 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42887
I20260619 09:04:07.461773 16651 heartbeater.cc:464] Registering TS with master...
I20260619 09:04:07.461942 16651 heartbeater.cc:511] Master 127.0.98.62:42887 requested a full tablet report, sending...
I20260619 09:04:07.462312 16325 ts_manager.cc:194] Registered new tserver with Master: 51ec1359afd84648a293aff68cab2a8e (127.0.98.2:33867)
I20260619 09:04:07.462661 16325 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:53217
W20260619 09:04:07.533052 16654 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 09:04:07.533244 16654 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 09:04:07.533267 16654 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 09:04:07.534404 16654 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 09:04:07.534441 16654 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 09:04:07.534466 16654 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 09:04:07.535887 16654 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859658134510-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859658134510-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859658134510-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:46739
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859658134510-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=38789
--tserver_master_addrs=127.0.98.62:42887
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16654
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859658134510-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 09:04:07.536092 16654 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 09:04:07.536290 16654 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 09:04:07.540980 16660 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:04:07.541003 16662 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:04:07.541126 16654 server_base.cc:1034] running on GCE node
W20260619 09:04:07.540982 16659 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:04:07.541369 16654 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 09:04:07.541589 16654 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 09:04:07.542750 16654 hybrid_clock.cc:648] HybridClock initialized: now 1781859847542730 us; error 27 us; skew 500 ppm
Jun 19 09:04:07 dist-test-slave-mtdr krb5kdc[15747](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859847, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 09:04:07.545147 16654 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 09:04:07.545594 16654 webserver.cc:460] Webserver started at http://127.0.98.3:38789/ using document root <none> and password file <none>
I20260619 09:04:07.545782 16654 fs_manager.cc:359] Metadata directory not provided
I20260619 09:04:07.545840 16654 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 09:04:07.547093 16654 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260619 09:04:07.547665 16669 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 09:04:07.547819 16654 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 09:04:07.547899 16654 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859658134510-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859658134510-392-0/minicluster-data/ts-2/wal
uuid: "ce39db3fe58a4151a4795ecaf0942bc9"
format_stamp: "Formatted at 2026-06-19 09:03:46 on dist-test-slave-mtdr"
server_key: "24578ab589f048f7a70103b4c4f8d7cd"
server_key_iv: "ef05717263c0ad09e7d552c0c4058654"
server_key_version: "encryptionkey@0"
I20260619 09:04:07.548207 16654 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859658134510-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859658134510-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859658134510-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 09:04:07.561790 16654 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 09:04:07.564049 16654 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 09:04:07.564208 16654 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 09:04:07.564422 16654 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 09:04:07.564706 16654 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 09:04:07.564742 16654 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 09:04:07.564765 16654 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 09:04:07.564778 16654 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 09:04:07.570673 16654 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:46739
I20260619 09:04:07.570708 16782 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:46739 every 8 connection(s)
I20260619 09:04:07.571077 16654 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859658134510-392-0/minicluster-data/ts-2/data/info.pb
Jun 19 09:04:07 dist-test-slave-mtdr krb5kdc[15747](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859847, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 09:04:07.578641 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ez5NB/build/release/bin/kudu as pid 16654
I20260619 09:04:07.580418 16516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:04:07.571780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50969 (local address 127.0.98.62:42887)
0619 09:04:07.571864 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0619 09:04:07.571866 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:04:07.572066 (+ 200us) server_negotiation.cc:407] Connection header received
0619 09:04:07.572578 (+ 512us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:04:07.572579 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:04:07.572589 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:04:07.572654 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:04:07.573676 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:04:07.574173 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:04:07.575334 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:04:07.575525 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:04:07.575584 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:04:07.577749 (+ 2165us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:04:07.577751 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:04:07.577751 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:04:07.577758 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:04:07.579216 (+ 1458us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:04:07.579571 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:04:07.579572 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:04:07.579573 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:04:07.579627 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:04:07.579838 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:04:07.579839 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:04:07.579839 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:04:07.579950 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:04:07.580004 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0619 09:04:07.580257 (+ 253us) server_negotiation.cc:299] Negotiation successful
0619 09:04:07.580332 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 09:04:07.580576 16785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:04:07.571759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42887 (local address 127.0.98.3:50969)
0619 09:04:07.571932 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0619 09:04:07.571968 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0619 09:04:07.572488 (+ 520us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:04:07.572760 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:04:07.572764 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:04:07.573389 (+ 625us) client_negotiation.cc:189] Negotiated authn=SASL
0619 09:04:07.573578 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:04:07.573582 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:04:07.574294 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:04:07.574295 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:04:07.575246 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:04:07.575249 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:04:07.575604 (+ 355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:04:07.575605 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:04:07.575728 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:04:07.576209 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 09:04:07.576218 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0619 09:04:07.577653 (+ 1435us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 09:04:07.579294 (+ 1641us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:04:07.579300 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:04:07.579303 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:04:07.579508 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:04:07.579676 (+ 168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:04:07.579677 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:04:07.579678 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:04:07.579785 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:04:07.580002 (+ 217us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 09:04:07.580008 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 09:04:07.580135 (+ 127us) client_negotiation.cc:769] Sending connection context
0619 09:04:07.580221 (+ 86us) client_negotiation.cc:240] Negotiation successful
0619 09:04:07.580371 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":84,"threads_started":1}
I20260619 09:04:07.581014 16783 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42887
I20260619 09:04:07.581110 16783 heartbeater.cc:464] Registering TS with master...
I20260619 09:04:07.581282 16783 heartbeater.cc:511] Master 127.0.98.62:42887 requested a full tablet report, sending...
I20260619 09:04:07.581653 16325 ts_manager.cc:194] Registered new tserver with Master: ce39db3fe58a4151a4795ecaf0942bc9 (127.0.98.3:46739)
I20260619 09:04:07.582122 16325 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:50969
I20260619 09:04:07.584983 16786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:04:07.579018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38314 (local address 127.0.98.62:42887)
0619 09:04:07.579117 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0619 09:04:07.579119 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:04:07.579208 (+ 89us) server_negotiation.cc:407] Connection header received
0619 09:04:07.579274 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:04:07.579274 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:04:07.579286 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:04:07.579340 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:04:07.579886 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:04:07.580354 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:04:07.581397 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:04:07.581659 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:04:07.581723 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:04:07.582928 (+ 1205us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:04:07.582929 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:04:07.582930 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:04:07.582935 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:04:07.584185 (+ 1250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:04:07.584470 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:04:07.584471 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:04:07.584471 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:04:07.584514 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:04:07.584629 (+ 115us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:04:07.584629 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:04:07.584629 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:04:07.584716 (+ 87us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:04:07.584832 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0619 09:04:07.584860 (+ 28us) server_negotiation.cc:299] Negotiation successful
0619 09:04:07.584913 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79,"thread_start_us":47,"threads_started":1}
I20260619 09:04:07.585569 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 09:04:07.588075 16786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:04:07.585846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38330 (local address 127.0.98.62:42887)
0619 09:04:07.585899 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0619 09:04:07.585901 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:04:07.585958 (+ 57us) server_negotiation.cc:407] Connection header received
0619 09:04:07.586013 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:04:07.586014 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:04:07.586040 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:04:07.586090 (+ 50us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 09:04:07.586165 (+ 75us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:04:07.586622 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:04:07.587384 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:04:07.587580 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:04:07.587643 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:04:07.587752 (+ 109us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 09:04:07.587788 (+ 36us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0619 09:04:07.588021 (+ 233us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:38330: token expired
Metrics: {"server-negotiator.queue_time_us":33}
W20260619 09:04:07.588068 16254 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.98.62:42887)
W20260619 09:04:07.588184 16786 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:38330: token expired
I20260619 09:04:07.593720 16786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:04:07.588674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38332 (local address 127.0.98.62:42887)
0619 09:04:07.588725 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0619 09:04:07.588726 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 09:04:07.588732 (+ 6us) server_negotiation.cc:407] Connection header received
0619 09:04:07.588748 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:04:07.588748 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:04:07.588754 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:04:07.588799 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:04:07.589203 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:04:07.589730 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:04:07.590371 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:04:07.590563 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:04:07.590613 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:04:07.591614 (+ 1001us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:04:07.591615 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:04:07.591616 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:04:07.591622 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:04:07.592706 (+ 1084us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:04:07.593045 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:04:07.593046 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:04:07.593046 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:04:07.593107 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:04:07.593251 (+ 144us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:04:07.593252 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:04:07.593252 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:04:07.593377 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:04:07.593455 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0619 09:04:07.593586 (+ 131us) server_negotiation.cc:299] Negotiation successful
0619 09:04:07.593647 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260619 09:04:07.594846 16325 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38332:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 09:04:07.596397 16325 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 09:04:07.598359 16429 tablet_service.cc:1467] Processing CreateTablet for tablet fc2a7f2001204f8eb75ce336ca95afa9 (DEFAULT_TABLE table=test-table [id=a6cce346c9794996a9f7bc5ac139cfb8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 09:04:07.598589 16429 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc2a7f2001204f8eb75ce336ca95afa9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 09:04:07.600101 16405 tablet_bootstrap.cc:492] T fc2a7f2001204f8eb75ce336ca95afa9 P c9252303f6c147b99263a5c7666a6134: Bootstrap starting.
I20260619 09:04:07.600414 16405 tablet_bootstrap.cc:654] T fc2a7f2001204f8eb75ce336ca95afa9 P c9252303f6c147b99263a5c7666a6134: Neither blocks nor log segments found. Creating new log.
I20260619 09:04:07.600915 16405 tablet_bootstrap.cc:492] T fc2a7f2001204f8eb75ce336ca95afa9 P c9252303f6c147b99263a5c7666a6134: No bootstrap required, opened a new log
I20260619 09:04:07.600960 16405 ts_tablet_manager.cc:1397] T fc2a7f2001204f8eb75ce336ca95afa9 P c9252303f6c147b99263a5c7666a6134: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260619 09:04:07.601058 16405 raft_consensus.cc:348] T fc2a7f2001204f8eb75ce336ca95afa9 P c9252303f6c147b99263a5c7666a6134 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce39db3fe58a4151a4795ecaf0942bc9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46739 } } peers { permanent_uuid: "51ec1359afd84648a293aff68cab2a8e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33867 } } peers { permanent_uuid: "c9252303f6c147b99263a5c7666a6134" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41245 } }
I20260619 09:04:07.601117 16405 raft_consensus.cc:374] T fc2a7f2001204f8eb75ce336ca95afa9 P c9252303f6c147b99263a5c7666a6134 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 09:04:07.601161 16405 raft_consensus.cc:729] T fc2a7f2001204f8eb75ce336ca95afa9 P c9252303f6c147b99263a5c7666a6134 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9252303f6c147b99263a5c7666a6134, State: Initialized, Role: FOLLOWER
I20260619 09:04:07.601208 16405 consensus_queue.cc:260] T fc2a7f2001204f8eb75ce336ca95afa9 P c9252303f6c147b99263a5c7666a6134 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce39db3fe58a4151a4795ecaf0942bc9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46739 } } peers { permanent_uuid: "51ec1359afd84648a293aff68cab2a8e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33867 } } peers { permanent_uuid: "c9252303f6c147b99263a5c7666a6134" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41245 } }
I20260619 09:04:07.601286 16405 ts_tablet_manager.cc:1428] T fc2a7f2001204f8eb75ce336ca95afa9 P c9252303f6c147b99263a5c7666a6134: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260619 09:04:07.601851 16789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:04:07.598127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33867 (local address 127.0.0.1:38294)
0619 09:04:07.598287 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0619 09:04:07.598297 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 09:04:07.598342 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:04:07.598691 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:04:07.598692 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:04:07.598695 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 09:04:07.598740 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:04:07.598741 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:04:07.599709 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:04:07.599710 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:04:07.600831 (+ 1121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:04:07.600833 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:04:07.601605 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:04:07.601606 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:04:07.601673 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:04:07.601680 (+ 7us) client_negotiation.cc:769] Sending connection context
0619 09:04:07.601735 (+ 55us) client_negotiation.cc:240] Negotiation successful
0619 09:04:07.601777 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"mutex_wait_us":20,"thread_start_us":107,"threads_started":1}
I20260619 09:04:07.602015 16520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:04:07.598100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46739 (local address 127.0.0.1:46862)
0619 09:04:07.598175 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0619 09:04:07.598186 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 09:04:07.598223 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:04:07.598724 (+ 501us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:04:07.598724 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:04:07.598725 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 09:04:07.598757 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:04:07.598758 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:04:07.599879 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:04:07.599880 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:04:07.600899 (+ 1019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:04:07.600901 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:04:07.601791 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:04:07.601792 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:04:07.601857 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:04:07.601860 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 09:04:07.601907 (+ 47us) client_negotiation.cc:240] Negotiation successful
0619 09:04:07.601950 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
I20260619 09:04:07.602246 16791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:04:07.598363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38294 (local address 127.0.98.2:33867)
0619 09:04:07.598519 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0619 09:04:07.598524 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 09:04:07.598536 (+ 12us) server_negotiation.cc:407] Connection header received
0619 09:04:07.598574 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:04:07.598576 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:04:07.598591 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:04:07.598835 (+ 244us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 09:04:07.598920 (+ 85us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:04:07.599639 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:04:07.600925 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:04:07.601538 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:04:07.601602 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:04:07.601664 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0619 09:04:07.602121 (+ 457us) server_negotiation.cc:299] Negotiation successful
0619 09:04:07.602167 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":68,"threads_started":1}
I20260619 09:04:07.602465 16790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:04:07.598320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46862 (local address 127.0.98.3:46739)
0619 09:04:07.598527 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0619 09:04:07.598531 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 09:04:07.598541 (+ 10us) server_negotiation.cc:407] Connection header received
0619 09:04:07.598578 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:04:07.598580 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:04:07.598594 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:04:07.598732 (+ 138us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 09:04:07.598810 (+ 78us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:04:07.599801 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:04:07.600973 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:04:07.601674 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:04:07.601742 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:04:07.601794 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0619 09:04:07.602348 (+ 554us) server_negotiation.cc:299] Negotiation successful
0619 09:04:07.602389 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":92,"threads_started":1}
I20260619 09:04:07.602929 16585 tablet_service.cc:1467] Processing CreateTablet for tablet fc2a7f2001204f8eb75ce336ca95afa9 (DEFAULT_TABLE table=test-table [id=a6cce346c9794996a9f7bc5ac139cfb8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 09:04:07.603075 16717 tablet_service.cc:1467] Processing CreateTablet for tablet fc2a7f2001204f8eb75ce336ca95afa9 (DEFAULT_TABLE table=test-table [id=a6cce346c9794996a9f7bc5ac139cfb8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 09:04:07.603263 16585 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc2a7f2001204f8eb75ce336ca95afa9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 09:04:07.603336 16717 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc2a7f2001204f8eb75ce336ca95afa9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 09:04:07.606887 16794 tablet_bootstrap.cc:492] T fc2a7f2001204f8eb75ce336ca95afa9 P ce39db3fe58a4151a4795ecaf0942bc9: Bootstrap starting.
I20260619 09:04:07.607465 16794 tablet_bootstrap.cc:654] T fc2a7f2001204f8eb75ce336ca95afa9 P ce39db3fe58a4151a4795ecaf0942bc9: Neither blocks nor log segments found. Creating new log.
I20260619 09:04:07.607966 16794 log.cc:824] T fc2a7f2001204f8eb75ce336ca95afa9 P ce39db3fe58a4151a4795ecaf0942bc9: Log is configured to *not* fsync() on all Append() calls
I20260619 09:04:07.606621 16793 tablet_bootstrap.cc:492] T fc2a7f2001204f8eb75ce336ca95afa9 P 51ec1359afd84648a293aff68cab2a8e: Bootstrap starting.
I20260619 09:04:07.608700 16793 tablet_bootstrap.cc:654] T fc2a7f2001204f8eb75ce336ca95afa9 P 51ec1359afd84648a293aff68cab2a8e: Neither blocks nor log segments found. Creating new log.
I20260619 09:04:07.608768 16794 tablet_bootstrap.cc:492] T fc2a7f2001204f8eb75ce336ca95afa9 P ce39db3fe58a4151a4795ecaf0942bc9: No bootstrap required, opened a new log
I20260619 09:04:07.608837 16794 ts_tablet_manager.cc:1397] T fc2a7f2001204f8eb75ce336ca95afa9 P ce39db3fe58a4151a4795ecaf0942bc9: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 09:04:07.609285 16793 log.cc:824] T fc2a7f2001204f8eb75ce336ca95afa9 P 51ec1359afd84648a293aff68cab2a8e: Log is configured to *not* fsync() on all Append() calls
I20260619 09:04:07.609974 16793 tablet_bootstrap.cc:492] T fc2a7f2001204f8eb75ce336ca95afa9 P 51ec1359afd84648a293aff68cab2a8e: No bootstrap required, opened a new log
I20260619 09:04:07.610040 16793 ts_tablet_manager.cc:1397] T fc2a7f2001204f8eb75ce336ca95afa9 P 51ec1359afd84648a293aff68cab2a8e: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260619 09:04:07.610690 16794 raft_consensus.cc:348] T fc2a7f2001204f8eb75ce336ca95afa9 P ce39db3fe58a4151a4795ecaf0942bc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce39db3fe58a4151a4795ecaf0942bc9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46739 } } peers { permanent_uuid: "51ec1359afd84648a293aff68cab2a8e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33867 } } peers { permanent_uuid: "c9252303f6c147b99263a5c7666a6134" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41245 } }
I20260619 09:04:07.610904 16794 raft_consensus.cc:374] T fc2a7f2001204f8eb75ce336ca95afa9 P ce39db3fe58a4151a4795ecaf0942bc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 09:04:07.610945 16794 raft_consensus.cc:729] T fc2a7f2001204f8eb75ce336ca95afa9 P ce39db3fe58a4151a4795ecaf0942bc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce39db3fe58a4151a4795ecaf0942bc9, State: Initialized, Role: FOLLOWER
I20260619 09:04:07.611074 16794 consensus_queue.cc:260] T fc2a7f2001204f8eb75ce336ca95afa9 P ce39db3fe58a4151a4795ecaf0942bc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce39db3fe58a4151a4795ecaf0942bc9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46739 } } peers { permanent_uuid: "51ec1359afd84648a293aff68cab2a8e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33867 } } peers { permanent_uuid: "c9252303f6c147b99263a5c7666a6134" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41245 } }
I20260619 09:04:07.611372 16794 ts_tablet_manager.cc:1428] T fc2a7f2001204f8eb75ce336ca95afa9 P ce39db3fe58a4151a4795ecaf0942bc9: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260619 09:04:07.611387 16783 heartbeater.cc:503] Master 127.0.98.62:42887 was elected leader, sending a full tablet report...
I20260619 09:04:07.611903 16793 raft_consensus.cc:348] T fc2a7f2001204f8eb75ce336ca95afa9 P 51ec1359afd84648a293aff68cab2a8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce39db3fe58a4151a4795ecaf0942bc9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46739 } } peers { permanent_uuid: "51ec1359afd84648a293aff68cab2a8e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33867 } } peers { permanent_uuid: "c9252303f6c147b99263a5c7666a6134" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41245 } }
I20260619 09:04:07.612051 16793 raft_consensus.cc:374] T fc2a7f2001204f8eb75ce336ca95afa9 P 51ec1359afd84648a293aff68cab2a8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 09:04:07.612083 16793 raft_consensus.cc:729] T fc2a7f2001204f8eb75ce336ca95afa9 P 51ec1359afd84648a293aff68cab2a8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51ec1359afd84648a293aff68cab2a8e, State: Initialized, Role: FOLLOWER
I20260619 09:04:07.612192 16793 consensus_queue.cc:260] T fc2a7f2001204f8eb75ce336ca95afa9 P 51ec1359afd84648a293aff68cab2a8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce39db3fe58a4151a4795ecaf0942bc9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46739 } } peers { permanent_uuid: "51ec1359afd84648a293aff68cab2a8e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33867 } } peers { permanent_uuid: "c9252303f6c147b99263a5c7666a6134" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41245 } }
I20260619 09:04:07.612433 16793 ts_tablet_manager.cc:1428] T fc2a7f2001204f8eb75ce336ca95afa9 P 51ec1359afd84648a293aff68cab2a8e: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260619 09:04:07.612473 16651 heartbeater.cc:503] Master 127.0.98.62:42887 was elected leader, sending a full tablet report...
W20260619 09:04:07.702462 16652 tablet.cc:2369] T fc2a7f2001204f8eb75ce336ca95afa9 P 51ec1359afd84648a293aff68cab2a8e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 09:04:07.811488 16514 tablet.cc:2369] T fc2a7f2001204f8eb75ce336ca95afa9 P c9252303f6c147b99263a5c7666a6134: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 09:04:07.821585 16784 tablet.cc:2369] T fc2a7f2001204f8eb75ce336ca95afa9 P ce39db3fe58a4151a4795ecaf0942bc9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 09:04:07.933157 16797 raft_consensus.cc:482] T fc2a7f2001204f8eb75ce336ca95afa9 P ce39db3fe58a4151a4795ecaf0942bc9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 09:04:07.933290 16797 raft_consensus.cc:504] T fc2a7f2001204f8eb75ce336ca95afa9 P ce39db3fe58a4151a4795ecaf0942bc9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce39db3fe58a4151a4795ecaf0942bc9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46739 } } peers { permanent_uuid: "51ec1359afd84648a293aff68cab2a8e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33867 } } peers { permanent_uuid: "c9252303f6c147b99263a5c7666a6134" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41245 } }
I20260619 09:04:07.933672 16797 leader_election.cc:288] T fc2a7f2001204f8eb75ce336ca95afa9 P ce39db3fe58a4151a4795ecaf0942bc9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 51ec1359afd84648a293aff68cab2a8e (127.0.98.2:33867), c9252303f6c147b99263a5c7666a6134 (127.0.98.1:41245)
I20260619 09:04:07.937593 16785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:04:07.933859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33867 (local address 127.0.98.3:48075)
0619 09:04:07.933923 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0619 09:04:07.933935 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 09:04:07.933978 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:04:07.934144 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:04:07.934144 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:04:07.934149 (+ 5us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 09:04:07.934222 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:04:07.934225 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:04:07.935236 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:04:07.935238 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:04:07.936502 (+ 1264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:04:07.936505 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:04:07.937290 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:04:07.937291 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:04:07.937383 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:04:07.937388 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 09:04:07.937451 (+ 63us) client_negotiation.cc:240] Negotiation successful
0619 09:04:07.937494 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43,"mutex_wait_us":70}
I20260619 09:04:07.937613 16791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:04:07.933923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48075 (local address 127.0.98.2:33867)
0619 09:04:07.933967 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0619 09:04:07.933969 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:04:07.933983 (+ 14us) server_negotiation.cc:407] Connection header received
0619 09:04:07.934037 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:04:07.934038 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:04:07.934050 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:04:07.934126 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 09:04:07.934340 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:04:07.935125 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:04:07.936614 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:04:07.937184 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:04:07.937278 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:04:07.937291 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0619 09:04:07.937491 (+ 200us) server_negotiation.cc:299] Negotiation successful
0619 09:04:07.937539 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260619 09:04:07.937601 16799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:04:07.933878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41245 (local address 127.0.98.3:36145)
0619 09:04:07.934022 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0619 09:04:07.934030 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0619 09:04:07.934068 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:04:07.934195 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:04:07.934196 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:04:07.934197 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 09:04:07.934253 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:04:07.934254 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:04:07.935259 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:04:07.935260 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:04:07.936502 (+ 1242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:04:07.936505 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:04:07.937350 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:04:07.937351 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:04:07.937407 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:04:07.937409 (+ 2us) client_negotiation.cc:769] Sending connection context
0619 09:04:07.937459 (+ 50us) client_negotiation.cc:240] Negotiation successful
0619 09:04:07.937494 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"mutex_wait_us":6,"thread_start_us":90,"threads_started":1}
I20260619 09:04:07.937614 16800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:04:07.933943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36145 (local address 127.0.98.1:41245)
0619 09:04:07.934062 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0619 09:04:07.934064 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:04:07.934072 (+ 8us) server_negotiation.cc:407] Connection header received
0619 09:04:07.934113 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:04:07.934113 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:04:07.934123 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:04:07.934183 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 09:04:07.934377 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:04:07.935125 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:04:07.936614 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:04:07.937229 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:04:07.937298 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:04:07.937311 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0619 09:04:07.937491 (+ 180us) server_negotiation.cc:299] Negotiation successful
0619 09:04:07.937540 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102,"thread_start_us":63,"threads_started":1}
I20260619 09:04:07.937980 16467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc2a7f2001204f8eb75ce336ca95afa9" candidate_uuid: "ce39db3fe58a4151a4795ecaf0942bc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9252303f6c147b99263a5c7666a6134" is_pre_election: true
I20260619 09:04:07.938154 16467 raft_consensus.cc:2454] T fc2a7f2001204f8eb75ce336ca95afa9 P c9252303f6c147b99263a5c7666a6134 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce39db3fe58a4151a4795ecaf0942bc9 in term 0.
I20260619 09:04:07.938186 16605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc2a7f2001204f8eb75ce336ca95afa9" candidate_uuid: "ce39db3fe58a4151a4795ecaf0942bc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51ec1359afd84648a293aff68cab2a8e" is_pre_election: true
I20260619 09:04:07.938341 16605 raft_consensus.cc:2454] T fc2a7f2001204f8eb75ce336ca95afa9 P 51ec1359afd84648a293aff68cab2a8e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce39db3fe58a4151a4795ecaf0942bc9 in term 0.
I20260619 09:04:07.938387 16671 leader_election.cc:302] T fc2a7f2001204f8eb75ce336ca95afa9 P ce39db3fe58a4151a4795ecaf0942bc9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c9252303f6c147b99263a5c7666a6134, ce39db3fe58a4151a4795ecaf0942bc9; no voters:
I20260619 09:04:07.938516 16797 raft_consensus.cc:2781] T fc2a7f2001204f8eb75ce336ca95afa9 P ce39db3fe58a4151a4795ecaf0942bc9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 09:04:07.938577 16797 raft_consensus.cc:482] T fc2a7f2001204f8eb75ce336ca95afa9 P ce39db3fe58a4151a4795ecaf0942bc9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 09:04:07.938601 16797 raft_consensus.cc:3037] T fc2a7f2001204f8eb75ce336ca95afa9 P ce39db3fe58a4151a4795ecaf0942bc9 [term 0 FOLLOWER]: Advancing to term 1
I20260619 09:04:07.939299 16797 raft_consensus.cc:504] T fc2a7f2001204f8eb75ce336ca95afa9 P ce39db3fe58a4151a4795ecaf0942bc9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce39db3fe58a4151a4795ecaf0942bc9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46739 } } peers { permanent_uuid: "51ec1359afd84648a293aff68cab2a8e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33867 } } peers { permanent_uuid: "c9252303f6c147b99263a5c7666a6134" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41245 } }
I20260619 09:04:07.939425 16797 leader_election.cc:288] T fc2a7f2001204f8eb75ce336ca95afa9 P ce39db3fe58a4151a4795ecaf0942bc9 [CANDIDATE]: Term 1 election: Requested vote from peers 51ec1359afd84648a293aff68cab2a8e (127.0.98.2:33867), c9252303f6c147b99263a5c7666a6134 (127.0.98.1:41245)
I20260619 09:04:07.939631 16605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc2a7f2001204f8eb75ce336ca95afa9" candidate_uuid: "ce39db3fe58a4151a4795ecaf0942bc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51ec1359afd84648a293aff68cab2a8e"
I20260619 09:04:07.939714 16605 raft_consensus.cc:3037] T fc2a7f2001204f8eb75ce336ca95afa9 P 51ec1359afd84648a293aff68cab2a8e [term 0 FOLLOWER]: Advancing to term 1
I20260619 09:04:07.940404 16605 raft_consensus.cc:2454] T fc2a7f2001204f8eb75ce336ca95afa9 P 51ec1359afd84648a293aff68cab2a8e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce39db3fe58a4151a4795ecaf0942bc9 in term 1.
I20260619 09:04:07.940508 16467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc2a7f2001204f8eb75ce336ca95afa9" candidate_uuid: "ce39db3fe58a4151a4795ecaf0942bc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9252303f6c147b99263a5c7666a6134"
I20260619 09:04:07.940577 16673 leader_election.cc:302] T fc2a7f2001204f8eb75ce336ca95afa9 P ce39db3fe58a4151a4795ecaf0942bc9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51ec1359afd84648a293aff68cab2a8e, ce39db3fe58a4151a4795ecaf0942bc9; no voters:
I20260619 09:04:07.940594 16467 raft_consensus.cc:3037] T fc2a7f2001204f8eb75ce336ca95afa9 P c9252303f6c147b99263a5c7666a6134 [term 0 FOLLOWER]: Advancing to term 1
I20260619 09:04:07.940665 16797 raft_consensus.cc:2781] T fc2a7f2001204f8eb75ce336ca95afa9 P ce39db3fe58a4151a4795ecaf0942bc9 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 09:04:07.940814 16797 raft_consensus.cc:686] T fc2a7f2001204f8eb75ce336ca95afa9 P ce39db3fe58a4151a4795ecaf0942bc9 [term 1 LEADER]: Becoming Leader. State: Replica: ce39db3fe58a4151a4795ecaf0942bc9, State: Running, Role: LEADER
I20260619 09:04:07.940903 16797 consensus_queue.cc:237] T fc2a7f2001204f8eb75ce336ca95afa9 P ce39db3fe58a4151a4795ecaf0942bc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce39db3fe58a4151a4795ecaf0942bc9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46739 } } peers { permanent_uuid: "51ec1359afd84648a293aff68cab2a8e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33867 } } peers { permanent_uuid: "c9252303f6c147b99263a5c7666a6134" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41245 } }
I20260619 09:04:07.941399 16467 raft_consensus.cc:2454] T fc2a7f2001204f8eb75ce336ca95afa9 P c9252303f6c147b99263a5c7666a6134 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce39db3fe58a4151a4795ecaf0942bc9 in term 1.
I20260619 09:04:07.941903 16325 catalog_manager.cc:5495] T fc2a7f2001204f8eb75ce336ca95afa9 P ce39db3fe58a4151a4795ecaf0942bc9 reported cstate change: term changed from 0 to 1, leader changed from <none> to ce39db3fe58a4151a4795ecaf0942bc9 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "ce39db3fe58a4151a4795ecaf0942bc9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce39db3fe58a4151a4795ecaf0942bc9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46739 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "51ec1359afd84648a293aff68cab2a8e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33867 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c9252303f6c147b99263a5c7666a6134" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41245 } health_report { overall_health: UNKNOWN } } }
I20260619 09:04:07.954317 16790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:04:07.952183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46874 (local address 127.0.98.3:46739)
0619 09:04:07.952229 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0619 09:04:07.952231 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:04:07.952239 (+ 8us) server_negotiation.cc:407] Connection header received
0619 09:04:07.952311 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:04:07.952313 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:04:07.952332 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:04:07.952393 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 09:04:07.952494 (+ 101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:04:07.952960 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:04:07.953642 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:04:07.953821 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:04:07.953879 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:04:07.954014 (+ 135us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 09:04:07.954108 (+ 94us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 09:04:07.954174 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0619 09:04:07.954209 (+ 35us) server_negotiation.cc:299] Negotiation successful
0619 09:04:07.954229 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260619 09:04:07.955942 16467 raft_consensus.cc:1261] T fc2a7f2001204f8eb75ce336ca95afa9 P c9252303f6c147b99263a5c7666a6134 [term 1 FOLLOWER]: Refusing update from remote peer ce39db3fe58a4151a4795ecaf0942bc9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 09:04:07.955943 16605 raft_consensus.cc:1261] T fc2a7f2001204f8eb75ce336ca95afa9 P 51ec1359afd84648a293aff68cab2a8e [term 1 FOLLOWER]: Refusing update from remote peer ce39db3fe58a4151a4795ecaf0942bc9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 09:04:07.956223 16801 consensus_queue.cc:1035] T fc2a7f2001204f8eb75ce336ca95afa9 P ce39db3fe58a4151a4795ecaf0942bc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9252303f6c147b99263a5c7666a6134" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41245 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 09:04:07.956365 16797 consensus_queue.cc:1035] T fc2a7f2001204f8eb75ce336ca95afa9 P ce39db3fe58a4151a4795ecaf0942bc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51ec1359afd84648a293aff68cab2a8e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 09:04:07.958285 16805 mvcc.cc:205] Tried to move back new op lower bound from 7298497937225199616 to 7298497937166544896. Current Snapshot: MvccSnapshot[applied={T|T < 7298497937225199616}]
I20260619 09:04:07.958858 16806 mvcc.cc:205] Tried to move back new op lower bound from 7298497937225199616 to 7298497937166544896. Current Snapshot: MvccSnapshot[applied={T|T < 7298497937225199616}]
I20260619 09:04:07.960971 16324 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38332:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 09:04:07.961050 16324 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38332:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 09:04:07.961709 16324 catalog_manager.cc:5782] T 00000000000000000000000000000000 P df0d2dbbe48b4c98a080b5e59d3c993d: Sending DeleteTablet for 3 replicas of tablet fc2a7f2001204f8eb75ce336ca95afa9
I20260619 09:04:07.962016 16429 tablet_service.cc:1514] Processing DeleteTablet for tablet fc2a7f2001204f8eb75ce336ca95afa9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 09:04:07 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36336
I20260619 09:04:07.962044 16717 tablet_service.cc:1514] Processing DeleteTablet for tablet fc2a7f2001204f8eb75ce336ca95afa9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 09:04:07 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46862
I20260619 09:04:07.962209 16585 tablet_service.cc:1514] Processing DeleteTablet for tablet fc2a7f2001204f8eb75ce336ca95afa9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 09:04:07 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38294
I20260619 09:04:07.962376 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ez5NB/build/release/bin/kudu with pid 16363
I20260619 09:04:07.964000 16815 tablet_replica.cc:330] T fc2a7f2001204f8eb75ce336ca95afa9 P 51ec1359afd84648a293aff68cab2a8e: stopping tablet replica
I20260619 09:04:07.964079 16815 raft_consensus.cc:2229] T fc2a7f2001204f8eb75ce336ca95afa9 P 51ec1359afd84648a293aff68cab2a8e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 09:04:07.964174 16815 raft_consensus.cc:2258] T fc2a7f2001204f8eb75ce336ca95afa9 P 51ec1359afd84648a293aff68cab2a8e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 09:04:07.964318 16813 tablet_replica.cc:330] T fc2a7f2001204f8eb75ce336ca95afa9 P ce39db3fe58a4151a4795ecaf0942bc9: stopping tablet replica
I20260619 09:04:07.964408 16813 raft_consensus.cc:2229] T fc2a7f2001204f8eb75ce336ca95afa9 P ce39db3fe58a4151a4795ecaf0942bc9 [term 1 LEADER]: Raft consensus shutting down.
I20260619 09:04:07.964519 16813 raft_consensus.cc:2258] T fc2a7f2001204f8eb75ce336ca95afa9 P ce39db3fe58a4151a4795ecaf0942bc9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 09:04:07.964730 16815 ts_tablet_manager.cc:1905] T fc2a7f2001204f8eb75ce336ca95afa9 P 51ec1359afd84648a293aff68cab2a8e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 09:04:07.964951 16813 ts_tablet_manager.cc:1905] T fc2a7f2001204f8eb75ce336ca95afa9 P ce39db3fe58a4151a4795ecaf0942bc9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 09:04:07.966507 16813 ts_tablet_manager.cc:1918] T fc2a7f2001204f8eb75ce336ca95afa9 P ce39db3fe58a4151a4795ecaf0942bc9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 09:04:07.966531 16815 ts_tablet_manager.cc:1918] T fc2a7f2001204f8eb75ce336ca95afa9 P 51ec1359afd84648a293aff68cab2a8e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 09:04:07.966619 16815 log.cc:1196] T fc2a7f2001204f8eb75ce336ca95afa9 P 51ec1359afd84648a293aff68cab2a8e: Deleting WAL directory at /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859658134510-392-0/minicluster-data/ts-1/wal/wals/fc2a7f2001204f8eb75ce336ca95afa9
I20260619 09:04:07.966660 16813 log.cc:1196] T fc2a7f2001204f8eb75ce336ca95afa9 P ce39db3fe58a4151a4795ecaf0942bc9: Deleting WAL directory at /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859658134510-392-0/minicluster-data/ts-2/wal/wals/fc2a7f2001204f8eb75ce336ca95afa9
I20260619 09:04:07.966928 16815 ts_tablet_manager.cc:1939] T fc2a7f2001204f8eb75ce336ca95afa9 P 51ec1359afd84648a293aff68cab2a8e: Deleting consensus metadata
I20260619 09:04:07.967133 16813 ts_tablet_manager.cc:1939] T fc2a7f2001204f8eb75ce336ca95afa9 P ce39db3fe58a4151a4795ecaf0942bc9: Deleting consensus metadata
I20260619 09:04:07.967311 16312 catalog_manager.cc:4841] TS 51ec1359afd84648a293aff68cab2a8e (127.0.98.2:33867): tablet fc2a7f2001204f8eb75ce336ca95afa9 (table test-table [id=a6cce346c9794996a9f7bc5ac139cfb8]) successfully deleted
I20260619 09:04:07.967566 16309 catalog_manager.cc:4841] TS ce39db3fe58a4151a4795ecaf0942bc9 (127.0.98.3:46739): tablet fc2a7f2001204f8eb75ce336ca95afa9 (table test-table [id=a6cce346c9794996a9f7bc5ac139cfb8]) successfully deleted
W20260619 09:04:07.968128 16310 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:41245 (error 108)
I20260619 09:04:07.968472 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ez5NB/build/release/bin/kudu with pid 16519
I20260619 09:04:07.968559 16520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:04:07.968434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41245 (local address 127.0.0.1:36348)
0619 09:04:07.968486 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0619 09:04:07.968525 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260619 09:04:07.970175 16310 catalog_manager.cc:4584] TS c9252303f6c147b99263a5c7666a6134 (127.0.98.1:41245): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet fc2a7f2001204f8eb75ce336ca95afa9: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41245: connect: Connection refused (error 111)
I20260619 09:04:07.973114 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ez5NB/build/release/bin/kudu with pid 16654
I20260619 09:04:07.977677 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ez5NB/build/release/bin/kudu with pid 16292
I20260619 09:04:27.982336 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ez5NB/build/release/bin/kudu
/tmp/dist-test-task4ez5NB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859658134510-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859658134510-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859658134510-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859658134510-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42887
--webserver_interface=127.0.98.62
--webserver_port=42285
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859658134510-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42887
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859658134510-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859658134510-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859658134510-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 09:04:28.060154 16817 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 09:04:28.060353 16817 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 09:04:28.060374 16817 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 09:04:28.061461 16817 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260619 09:04:28.061496 16817 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260619 09:04:28.061506 16817 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260619 09:04:28.061515 16817 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 09:04:28.061528 16817 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 09:04:28.061539 16817 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 09:04:28.061549 16817 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 09:04:28.061559 16817 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 09:04:28.062979 16817 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859658134510-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859658134510-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42887
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859658134510-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42887
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859658134510-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=42285
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16817
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859658134510-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 09:04:28.063171 16817 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 09:04:28.063333 16817 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 09:04:28.068025 16822 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:04:28.068077 16817 server_base.cc:1034] running on GCE node
W20260619 09:04:28.068131 16823 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:04:28.068149 16825 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:04:28.068464 16817 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 09:04:28.068750 16817 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 09:04:28.069906 16817 hybrid_clock.cc:648] HybridClock initialized: now 1781859868069893 us; error 27 us; skew 500 ppm
Jun 19 09:04:28 dist-test-slave-mtdr krb5kdc[15747](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859868, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 09:04:28.075145 16817 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 09:04:28.075552 16817 webserver.cc:460] Webserver started at http://127.0.98.62:42285/ using document root <none> and password file <none>
I20260619 09:04:28.075742 16817 fs_manager.cc:359] Metadata directory not provided
I20260619 09:04:28.075789 16817 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 09:04:28.077087 16817 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 09:04:28.077657 16832 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 09:04:28.077818 16817 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 09:04:28.077896 16817 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859658134510-392-0/minicluster-data/master-0/data,/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859658134510-392-0/minicluster-data/master-0/wal
uuid: "df0d2dbbe48b4c98a080b5e59d3c993d"
format_stamp: "Formatted at 2026-06-19 09:03:46 on dist-test-slave-mtdr"
server_key: "2de344460d2ca414a0dcf14bc1f4db7b"
server_key_iv: "8dfc905c00a66b785545414fb96c3160"
server_key_version: "encryptionkey@0"
I20260619 09:04:28.078219 16817 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859658134510-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859658134510-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859658134510-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 09:04:28.084673 16817 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 09:04:28.086346 16817 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 09:04:28.086463 16817 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 09:04:28.090044 16817 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42887
I20260619 09:04:28.090081 16884 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42887 every 8 connection(s)
I20260619 09:04:28.090404 16817 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859658134510-392-0/minicluster-data/master-0/data/info.pb
I20260619 09:04:28.091795 16885 sys_catalog.cc:263] Verifying existing consensus state
I20260619 09:04:28.092396 16885 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df0d2dbbe48b4c98a080b5e59d3c993d: Bootstrap starting.
I20260619 09:04:28.097029 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ez5NB/build/release/bin/kudu as pid 16817
I20260619 09:04:28.097287 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ez5NB/build/release/bin/kudu
/tmp/dist-test-task4ez5NB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859658134510-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859658134510-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859658134510-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859658134510-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:41245
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:42887
--webserver_port=34623
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859658134510-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859658134510-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859658134510-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859658134510-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 09:04:28.100064 16885 log.cc:824] T 00000000000000000000000000000000 P df0d2dbbe48b4c98a080b5e59d3c993d: Log is configured to *not* fsync() on all Append() calls
I20260619 09:04:28.101982 16885 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df0d2dbbe48b4c98a080b5e59d3c993d: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 09:04:28.102231 16885 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df0d2dbbe48b4c98a080b5e59d3c993d: Bootstrap complete.
I20260619 09:04:28.104285 16885 raft_consensus.cc:348] T 00000000000000000000000000000000 P df0d2dbbe48b4c98a080b5e59d3c993d [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df0d2dbbe48b4c98a080b5e59d3c993d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42887 } }
I20260619 09:04:28.104486 16885 raft_consensus.cc:729] T 00000000000000000000000000000000 P df0d2dbbe48b4c98a080b5e59d3c993d [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: df0d2dbbe48b4c98a080b5e59d3c993d, State: Initialized, Role: FOLLOWER
I20260619 09:04:28.104619 16885 consensus_queue.cc:260] T 00000000000000000000000000000000 P df0d2dbbe48b4c98a080b5e59d3c993d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df0d2dbbe48b4c98a080b5e59d3c993d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42887 } }
I20260619 09:04:28.104672 16885 raft_consensus.cc:388] T 00000000000000000000000000000000 P df0d2dbbe48b4c98a080b5e59d3c993d [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 09:04:28.104707 16885 raft_consensus.cc:482] T 00000000000000000000000000000000 P df0d2dbbe48b4c98a080b5e59d3c993d [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 09:04:28.104751 16885 raft_consensus.cc:3037] T 00000000000000000000000000000000 P df0d2dbbe48b4c98a080b5e59d3c993d [term 2 FOLLOWER]: Advancing to term 3
I20260619 09:04:28.105629 16885 raft_consensus.cc:504] T 00000000000000000000000000000000 P df0d2dbbe48b4c98a080b5e59d3c993d [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df0d2dbbe48b4c98a080b5e59d3c993d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42887 } }
I20260619 09:04:28.105710 16885 leader_election.cc:302] T 00000000000000000000000000000000 P df0d2dbbe48b4c98a080b5e59d3c993d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: df0d2dbbe48b4c98a080b5e59d3c993d; no voters:
I20260619 09:04:28.105846 16885 leader_election.cc:288] T 00000000000000000000000000000000 P df0d2dbbe48b4c98a080b5e59d3c993d [CANDIDATE]: Term 3 election: Requested vote from peers
I20260619 09:04:28.105895 16889 raft_consensus.cc:2781] T 00000000000000000000000000000000 P df0d2dbbe48b4c98a080b5e59d3c993d [term 3 FOLLOWER]: Leader election won for term 3
I20260619 09:04:28.106006 16889 raft_consensus.cc:686] T 00000000000000000000000000000000 P df0d2dbbe48b4c98a080b5e59d3c993d [term 3 LEADER]: Becoming Leader. State: Replica: df0d2dbbe48b4c98a080b5e59d3c993d, State: Running, Role: LEADER
I20260619 09:04:28.106092 16885 sys_catalog.cc:564] T 00000000000000000000000000000000 P df0d2dbbe48b4c98a080b5e59d3c993d [sys.catalog]: configured and running, proceeding with master startup.
I20260619 09:04:28.106086 16889 consensus_queue.cc:237] T 00000000000000000000000000000000 P df0d2dbbe48b4c98a080b5e59d3c993d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df0d2dbbe48b4c98a080b5e59d3c993d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42887 } }
I20260619 09:04:28.106489 16890 sys_catalog.cc:455] T 00000000000000000000000000000000 P df0d2dbbe48b4c98a080b5e59d3c993d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "df0d2dbbe48b4c98a080b5e59d3c993d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df0d2dbbe48b4c98a080b5e59d3c993d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42887 } } }
I20260619 09:04:28.106572 16890 sys_catalog.cc:458] T 00000000000000000000000000000000 P df0d2dbbe48b4c98a080b5e59d3c993d [sys.catalog]: This master's current role is: LEADER
I20260619 09:04:28.106854 16891 sys_catalog.cc:455] T 00000000000000000000000000000000 P df0d2dbbe48b4c98a080b5e59d3c993d [sys.catalog]: SysCatalogTable state changed. Reason: New leader df0d2dbbe48b4c98a080b5e59d3c993d. Latest consensus state: current_term: 3 leader_uuid: "df0d2dbbe48b4c98a080b5e59d3c993d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df0d2dbbe48b4c98a080b5e59d3c993d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42887 } } }
I20260619 09:04:28.106920 16891 sys_catalog.cc:458] T 00000000000000000000000000000000 P df0d2dbbe48b4c98a080b5e59d3c993d [sys.catalog]: This master's current role is: LEADER
I20260619 09:04:28.107038 16895 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 09:04:28.107785 16895 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 09:04:28.107885 16895 catalog_manager.cc:1259] Loaded cluster ID: fb77a7cd21ee4af78f446a5fba7eb31e
I20260619 09:04:28.107916 16895 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 09:04:28.108727 16895 catalog_manager.cc:1504] Loading token signing keys...
I20260619 09:04:28.108880 16895 catalog_manager.cc:5879] T 00000000000000000000000000000000 P df0d2dbbe48b4c98a080b5e59d3c993d: Loaded TSK: 0
I20260619 09:04:28.108914 16895 catalog_manager.cc:5879] T 00000000000000000000000000000000 P df0d2dbbe48b4c98a080b5e59d3c993d: Loaded TSK: 1
F20260619 09:04:28.109105 16895 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f734ec4c9ad google::LogMessage::Fail() at ??:0
@ 0x7f734ec4f88a google::LogMessage::SendToLog() at ??:0
@ 0x7f734ec4c4de google::LogMessage::Flush() at ??:0
@ 0x7f734ec4da19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f73513b9104 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f73513b6d41 kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f7356ba61cd kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f7356ba648e kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f7356ba6594 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7f7356b6cc48 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7f7356b76ad0 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f7350b1a0d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f7350b0f954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f73519736db start_thread at ??:0
@ 0x7f734d74a61f clone at ??:0
W20260619 09:04:28.193769 16887 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 09:04:28.193948 16887 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 09:04:28.193969 16887 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 09:04:28.195089 16887 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 09:04:28.195127 16887 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 09:04:28.195152 16887 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 09:04:28.196532 16887 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859658134510-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859658134510-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859658134510-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:41245
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859658134510-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=34623
--tserver_master_addrs=127.0.98.62:42887
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16887
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859658134510-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 09:04:28.196779 16887 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 09:04:28.197038 16887 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 09:04:28.201619 16926 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:04:28.201670 16887 server_base.cc:1034] running on GCE node
W20260619 09:04:28.201642 16929 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:04:28.201767 16927 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:04:28.201977 16887 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 09:04:28.202185 16887 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 09:04:28.203325 16887 hybrid_clock.cc:648] HybridClock initialized: now 1781859868203294 us; error 43 us; skew 500 ppm
Jun 19 09:04:28 dist-test-slave-mtdr krb5kdc[15747](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859868, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 09:04:28.205663 16887 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 09:04:28.206070 16887 webserver.cc:460] Webserver started at http://127.0.98.1:34623/ using document root <none> and password file <none>
I20260619 09:04:28.206261 16887 fs_manager.cc:359] Metadata directory not provided
I20260619 09:04:28.206310 16887 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 09:04:28.207623 16887 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.000s
I20260619 09:04:28.208243 16936 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 09:04:28.208389 16887 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 09:04:28.208467 16887 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859658134510-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859658134510-392-0/minicluster-data/ts-0/wal
uuid: "c9252303f6c147b99263a5c7666a6134"
format_stamp: "Formatted at 2026-06-19 09:03:46 on dist-test-slave-mtdr"
server_key: "4c23a7631a8101e8f8f522d0837b3dc8"
server_key_iv: "ff9c47905da211813b68b95dda196455"
server_key_version: "encryptionkey@0"
I20260619 09:04:28.208760 16887 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859658134510-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859658134510-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859658134510-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 09:04:28.223423 16887 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 09:04:28.225481 16887 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 09:04:28.225611 16887 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 09:04:28.225798 16887 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 09:04:28.226188 16943 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260619 09:04:28.228066 16887 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260619 09:04:28.228132 16887 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260619 09:04:28.228171 16887 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260619 09:04:28.228797 16887 ts_tablet_manager.cc:610] Registered 1 tablets
I20260619 09:04:28.228832 16887 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260619 09:04:28.228921 16943 tablet_bootstrap.cc:492] T fc2a7f2001204f8eb75ce336ca95afa9 P c9252303f6c147b99263a5c7666a6134: Bootstrap starting.
I20260619 09:04:28.235857 16887 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41245
I20260619 09:04:28.236006 17050 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41245 every 8 connection(s)
I20260619 09:04:28.236258 16887 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859658134510-392-0/minicluster-data/ts-0/data/info.pb
I20260619 09:04:28.239853 16943 log.cc:824] T fc2a7f2001204f8eb75ce336ca95afa9 P c9252303f6c147b99263a5c7666a6134: Log is configured to *not* fsync() on all Append() calls
I20260619 09:04:28.241910 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ez5NB/build/release/bin/kudu as pid 16887
I20260619 09:04:28.242192 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ez5NB/build/release/bin/kudu
/tmp/dist-test-task4ez5NB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859658134510-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859658134510-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859658134510-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859658134510-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:33867
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:42887
--webserver_port=36909
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859658134510-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859658134510-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859658134510-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859658134510-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 09:04:28.242419 16943 tablet_bootstrap.cc:492] T fc2a7f2001204f8eb75ce336ca95afa9 P c9252303f6c147b99263a5c7666a6134: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 09:04:28.242712 16943 tablet_bootstrap.cc:492] T fc2a7f2001204f8eb75ce336ca95afa9 P c9252303f6c147b99263a5c7666a6134: Bootstrap complete.
I20260619 09:04:28.243757 16943 ts_tablet_manager.cc:1397] T fc2a7f2001204f8eb75ce336ca95afa9 P c9252303f6c147b99263a5c7666a6134: Time spent bootstrapping tablet: real 0.015s user 0.001s sys 0.012s
I20260619 09:04:28.245151 16943 raft_consensus.cc:348] T fc2a7f2001204f8eb75ce336ca95afa9 P c9252303f6c147b99263a5c7666a6134 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce39db3fe58a4151a4795ecaf0942bc9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46739 } } peers { permanent_uuid: "51ec1359afd84648a293aff68cab2a8e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33867 } } peers { permanent_uuid: "c9252303f6c147b99263a5c7666a6134" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41245 } }
I20260619 09:04:28.245507 16943 raft_consensus.cc:729] T fc2a7f2001204f8eb75ce336ca95afa9 P c9252303f6c147b99263a5c7666a6134 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9252303f6c147b99263a5c7666a6134, State: Initialized, Role: FOLLOWER
I20260619 09:04:28.245672 16943 consensus_queue.cc:260] T fc2a7f2001204f8eb75ce336ca95afa9 P c9252303f6c147b99263a5c7666a6134 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce39db3fe58a4151a4795ecaf0942bc9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46739 } } peers { permanent_uuid: "51ec1359afd84648a293aff68cab2a8e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33867 } } peers { permanent_uuid: "c9252303f6c147b99263a5c7666a6134" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41245 } }
I20260619 09:04:28.245944 16943 ts_tablet_manager.cc:1428] T fc2a7f2001204f8eb75ce336ca95afa9 P c9252303f6c147b99263a5c7666a6134: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
W20260619 09:04:28.287868 17053 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 09:04:28.236890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42887 (local address 127.0.98.1:58619)
0619 09:04:28.237067 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0619 09:04:28.237101 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0619 09:04:28.237533 (+ 432us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:04:28.287692 (+ 50159us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42887: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":97,"threads_started":1}
W20260619 09:04:28.288120 17051 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:42887 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:42887: Client connection negotiation failed: client connection to 127.0.98.62:42887: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260619 09:04:28.288458 17053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:04:28.288355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42887 (local address 127.0.98.1:34099)
0619 09:04:28.288399 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0619 09:04:28.288437 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260619 09:04:28.288759 17053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:04:28.288675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42887 (local address 127.0.98.1:37943)
0619 09:04:28.288716 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0619 09:04:28.288738 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260619 09:04:28.288861 17051 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 09:04:28.323567 17055 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 09:04:28.323758 17055 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 09:04:28.323781 17055 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 09:04:28.324939 17055 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 09:04:28.324976 17055 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 09:04:28.325001 17055 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 09:04:28.326359 17055 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859658134510-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859658134510-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859658134510-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:33867
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859658134510-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=36909
--tserver_master_addrs=127.0.98.62:42887
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17055
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859658134510-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 09:04:28.326542 17055 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 09:04:28.326706 17055 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 09:04:28.331357 17061 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:04:28.331422 17064 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:04:28.331454 17062 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:04:28.331696 17055 server_base.cc:1034] running on GCE node
I20260619 09:04:28.331892 17055 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 09:04:28.332120 17055 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 09:04:28.333272 17055 hybrid_clock.cc:648] HybridClock initialized: now 1781859868333248 us; error 33 us; skew 500 ppm
Jun 19 09:04:28 dist-test-slave-mtdr krb5kdc[15747](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859868, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 09:04:28.335595 17055 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 09:04:28.336010 17055 webserver.cc:460] Webserver started at http://127.0.98.2:36909/ using document root <none> and password file <none>
I20260619 09:04:28.336206 17055 fs_manager.cc:359] Metadata directory not provided
I20260619 09:04:28.336253 17055 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 09:04:28.337539 17055 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 09:04:28.338114 17071 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 09:04:28.338270 17055 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 09:04:28.338351 17055 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859658134510-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859658134510-392-0/minicluster-data/ts-1/wal
uuid: "51ec1359afd84648a293aff68cab2a8e"
format_stamp: "Formatted at 2026-06-19 09:03:46 on dist-test-slave-mtdr"
server_key: "3c6eeecd77c8caf9312c2c16884bba17"
server_key_iv: "5bce3236820d4114c2c8a452b1ee9625"
server_key_version: "encryptionkey@0"
I20260619 09:04:28.338670 17055 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859658134510-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859658134510-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859658134510-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 09:04:28.350039 17055 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 09:04:28.352038 17055 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 09:04:28.352169 17055 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 09:04:28.352363 17055 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 09:04:28.352643 17055 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 09:04:28.352676 17055 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 09:04:28.352703 17055 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 09:04:28.352720 17055 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 09:04:28.358103 17055 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33867
I20260619 09:04:28.358222 17184 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33867 every 8 connection(s)
I20260619 09:04:28.358472 17055 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859658134510-392-0/minicluster-data/ts-1/data/info.pb
I20260619 09:04:28.359578 17187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:04:28.359231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42887 (local address 127.0.98.2:45729)
0619 09:04:28.359376 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0619 09:04:28.359443 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":78,"threads_started":1}
W20260619 09:04:28.359977 17185 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:42887 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:42887: Client connection negotiation failed: client connection to 127.0.98.62:42887: connect: Connection refused (error 111)
I20260619 09:04:28.360252 17187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:04:28.360173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42887 (local address 127.0.98.2:45191)
0619 09:04:28.360209 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0619 09:04:28.360235 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
I20260619 09:04:28.360512 17187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:04:28.360444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42887 (local address 127.0.98.2:40031)
0619 09:04:28.360477 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0619 09:04:28.360498 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
W20260619 09:04:28.360599 17185 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260619 09:04:28.368431 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ez5NB/build/release/bin/kudu as pid 17055
I20260619 09:04:28.368651 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ez5NB/build/release/bin/kudu
/tmp/dist-test-task4ez5NB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859658134510-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859658134510-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859658134510-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859658134510-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:46739
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:42887
--webserver_port=38789
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859658134510-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859658134510-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859658134510-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859658134510-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 09:04:28.446502 17188 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 09:04:28.446692 17188 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 09:04:28.446712 17188 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 09:04:28.447818 17188 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 09:04:28.447861 17188 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 09:04:28.447894 17188 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 09:04:28.449247 17188 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859658134510-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859658134510-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859658134510-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:46739
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859658134510-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=38789
--tserver_master_addrs=127.0.98.62:42887
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17188
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859658134510-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 09:04:28.449427 17188 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 09:04:28.449592 17188 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 09:04:28.454479 17193 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:04:28.454566 17194 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:04:28.454569 17196 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:04:28.454674 17188 server_base.cc:1034] running on GCE node
I20260619 09:04:28.454881 17188 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 09:04:28.455157 17188 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 09:04:28.456341 17188 hybrid_clock.cc:648] HybridClock initialized: now 1781859868456324 us; error 27 us; skew 500 ppm
Jun 19 09:04:28 dist-test-slave-mtdr krb5kdc[15747](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859868, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 09:04:28.458600 17188 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 09:04:28.459069 17188 webserver.cc:460] Webserver started at http://127.0.98.3:38789/ using document root <none> and password file <none>
I20260619 09:04:28.459273 17188 fs_manager.cc:359] Metadata directory not provided
I20260619 09:04:28.459321 17188 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 09:04:28.460556 17188 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 09:04:28.461140 17203 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 09:04:28.461292 17188 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 09:04:28.461369 17188 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859658134510-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859658134510-392-0/minicluster-data/ts-2/wal
uuid: "ce39db3fe58a4151a4795ecaf0942bc9"
format_stamp: "Formatted at 2026-06-19 09:03:46 on dist-test-slave-mtdr"
server_key: "24578ab589f048f7a70103b4c4f8d7cd"
server_key_iv: "ef05717263c0ad09e7d552c0c4058654"
server_key_version: "encryptionkey@0"
I20260619 09:04:28.461699 17188 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859658134510-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859658134510-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859658134510-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 09:04:28.489442 17188 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 09:04:28.491206 17188 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 09:04:28.491339 17188 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 09:04:28.491564 17188 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 09:04:28.491839 17188 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 09:04:28.491871 17188 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 09:04:28.491899 17188 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 09:04:28.491915 17188 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 09:04:28.497359 17188 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:46739
I20260619 09:04:28.497407 17316 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:46739 every 8 connection(s)
I20260619 09:04:28.497730 17188 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781859658134510-392-0/minicluster-data/ts-2/data/info.pb
I20260619 09:04:28.498703 17319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:04:28.498351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42887 (local address 127.0.98.3:50989)
0619 09:04:28.498486 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0619 09:04:28.498567 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":65,"threads_started":1}
W20260619 09:04:28.499082 17317 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:42887 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:42887: Client connection negotiation failed: client connection to 127.0.98.62:42887: connect: Connection refused (error 111)
I20260619 09:04:28.499310 17319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:04:28.499232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42887 (local address 127.0.98.3:44265)
0619 09:04:28.499270 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0619 09:04:28.499294 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260619 09:04:28.499521 17319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:04:28.499456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42887 (local address 127.0.98.3:58087)
0619 09:04:28.499483 (+ 27us) negotiation.cc:107] Waiting for socket to connect
0619 09:04:28.499510 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
W20260619 09:04:28.499598 17317 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260619 09:04:28.503311 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ez5NB/build/release/bin/kudu as pid 17188
I20260619 09:04:29.289398 17321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:04:29.289188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42887 (local address 127.0.98.1:55997)
0619 09:04:29.289301 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0619 09:04:29.289336 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":59,"threads_started":1}
I20260619 09:04:29.361114 17322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:04:29.360878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42887 (local address 127.0.98.2:56011)
0619 09:04:29.361006 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0619 09:04:29.361038 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":53,"threads_started":1}
I20260619 09:04:29.500124 17323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:04:29.499913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42887 (local address 127.0.98.3:60335)
0619 09:04:29.500042 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0619 09:04:29.500076 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":64,"threads_started":1}
I20260619 09:04:29.569332 17324 raft_consensus.cc:482] T fc2a7f2001204f8eb75ce336ca95afa9 P c9252303f6c147b99263a5c7666a6134 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 09:04:29.569456 17324 raft_consensus.cc:504] T fc2a7f2001204f8eb75ce336ca95afa9 P c9252303f6c147b99263a5c7666a6134 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce39db3fe58a4151a4795ecaf0942bc9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46739 } } peers { permanent_uuid: "51ec1359afd84648a293aff68cab2a8e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33867 } } peers { permanent_uuid: "c9252303f6c147b99263a5c7666a6134" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41245 } }
I20260619 09:04:29.569733 17324 leader_election.cc:288] T fc2a7f2001204f8eb75ce336ca95afa9 P c9252303f6c147b99263a5c7666a6134 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ce39db3fe58a4151a4795ecaf0942bc9 (127.0.98.3:46739), 51ec1359afd84648a293aff68cab2a8e (127.0.98.2:33867)
Jun 19 09:04:29 dist-test-slave-mtdr krb5kdc[15747](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859868, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.3@KRBTEST.COM
Jun 19 09:04:29 dist-test-slave-mtdr krb5kdc[15747](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859868, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
I20260619 09:04:29.579159 17321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:04:29.569874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46739 (local address 127.0.98.1:48823)
0619 09:04:29.569927 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0619 09:04:29.569938 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 09:04:29.569993 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:04:29.570801 (+ 808us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:04:29.570803 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:04:29.571221 (+ 418us) client_negotiation.cc:189] Negotiated authn=SASL
0619 09:04:29.571394 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:04:29.571399 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:04:29.572438 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:04:29.572439 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:04:29.573222 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:04:29.573224 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:04:29.573767 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:04:29.573768 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:04:29.573877 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:04:29.574424 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 09:04:29.574428 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 09:04:29.575820 (+ 1392us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 09:04:29.577528 (+ 1708us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:04:29.577534 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:04:29.577536 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:04:29.577800 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:04:29.578107 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:04:29.578108 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:04:29.578108 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:04:29.578203 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:04:29.578609 (+ 406us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 09:04:29.578616 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 09:04:29.578760 (+ 144us) client_negotiation.cc:769] Sending connection context
0619 09:04:29.578884 (+ 124us) client_negotiation.cc:240] Negotiation successful
0619 09:04:29.579042 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32,"mutex_wait_us":51}
I20260619 09:04:29.579211 17325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:04:29.569957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48823 (local address 127.0.98.3:46739)
0619 09:04:29.570101 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0619 09:04:29.570114 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 09:04:29.570145 (+ 31us) server_negotiation.cc:407] Connection header received
0619 09:04:29.570604 (+ 459us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:04:29.570606 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:04:29.570642 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:04:29.570766 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:04:29.571509 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:04:29.572339 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:04:29.573306 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:04:29.573690 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:04:29.573784 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:04:29.575914 (+ 2130us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:04:29.575918 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:04:29.575918 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:04:29.575927 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:04:29.577447 (+ 1520us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:04:29.577905 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:04:29.577908 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:04:29.577908 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:04:29.578034 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:04:29.578262 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:04:29.578263 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:04:29.578263 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:04:29.578547 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:04:29.578615 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0619 09:04:29.578950 (+ 335us) server_negotiation.cc:299] Negotiation successful
0619 09:04:29.579083 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":45,"threads_started":1}
I20260619 09:04:29.579403 17327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:04:29.569995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36441 (local address 127.0.98.2:33867)
0619 09:04:29.570144 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0619 09:04:29.570151 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 09:04:29.570179 (+ 28us) server_negotiation.cc:407] Connection header received
0619 09:04:29.570578 (+ 399us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:04:29.570580 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:04:29.570620 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:04:29.570727 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:04:29.571514 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:04:29.572336 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:04:29.573308 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:04:29.573690 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:04:29.573789 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:04:29.576268 (+ 2479us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:04:29.576272 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:04:29.576272 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:04:29.576284 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:04:29.577930 (+ 1646us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:04:29.578279 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:04:29.578281 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:04:29.578282 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:04:29.578358 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:04:29.578563 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:04:29.578563 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:04:29.578564 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:04:29.578839 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:04:29.578904 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0619 09:04:29.579086 (+ 182us) server_negotiation.cc:299] Negotiation successful
0619 09:04:29.579200 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":62,"threads_started":1}
I20260619 09:04:29.579242 17326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:04:29.569878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33867 (local address 127.0.98.1:36441)
0619 09:04:29.570084 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0619 09:04:29.570093 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 09:04:29.570145 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:04:29.570801 (+ 656us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:04:29.570804 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:04:29.571216 (+ 412us) client_negotiation.cc:189] Negotiated authn=SASL
0619 09:04:29.571394 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:04:29.571399 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:04:29.572428 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:04:29.572429 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:04:29.573223 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:04:29.573225 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:04:29.573769 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:04:29.573770 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:04:29.573877 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:04:29.574371 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 09:04:29.574378 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0619 09:04:29.576181 (+ 1803us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 09:04:29.578013 (+ 1832us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:04:29.578014 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:04:29.578015 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:04:29.578209 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:04:29.578416 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:04:29.578417 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:04:29.578417 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:04:29.578503 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:04:29.578905 (+ 402us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 09:04:29.578906 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 09:04:29.578955 (+ 49us) client_negotiation.cc:769] Sending connection context
0619 09:04:29.579021 (+ 66us) client_negotiation.cc:240] Negotiation successful
0619 09:04:29.579105 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":195,"mutex_wait_us":90,"thread_start_us":95,"threads_started":1}
W20260619 09:04:29.580440 16937 leader_election.cc:341] T fc2a7f2001204f8eb75ce336ca95afa9 P c9252303f6c147b99263a5c7666a6134 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ce39db3fe58a4151a4795ecaf0942bc9 (127.0.98.3:46739): Not found: Tablet not found: fc2a7f2001204f8eb75ce336ca95afa9
W20260619 09:04:29.580529 16940 leader_election.cc:341] T fc2a7f2001204f8eb75ce336ca95afa9 P c9252303f6c147b99263a5c7666a6134 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 51ec1359afd84648a293aff68cab2a8e (127.0.98.2:33867): Not found: Tablet not found: fc2a7f2001204f8eb75ce336ca95afa9
I20260619 09:04:29.580574 16940 leader_election.cc:302] T fc2a7f2001204f8eb75ce336ca95afa9 P c9252303f6c147b99263a5c7666a6134 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c9252303f6c147b99263a5c7666a6134; no voters: 51ec1359afd84648a293aff68cab2a8e, ce39db3fe58a4151a4795ecaf0942bc9
I20260619 09:04:29.580667 17324 raft_consensus.cc:2726] T fc2a7f2001204f8eb75ce336ca95afa9 P c9252303f6c147b99263a5c7666a6134 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 09:04:30.290006 17328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:04:30.289809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42887 (local address 127.0.98.1:59241)
0619 09:04:30.289925 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0619 09:04:30.289959 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":60,"threads_started":1}
I20260619 09:04:30.361769 17329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:04:30.361552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42887 (local address 127.0.98.2:37319)
0619 09:04:30.361687 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0619 09:04:30.361728 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":57,"threads_started":1}
I20260619 09:04:30.500691 17330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:04:30.500505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42887 (local address 127.0.98.3:48551)
0619 09:04:30.500619 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0619 09:04:30.500653 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":63,"threads_started":1}
I20260619 09:04:31.290632 17331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:04:31.290427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42887 (local address 127.0.98.1:54331)
0619 09:04:31.290552 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0619 09:04:31.290585 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":58,"threads_started":1}
I20260619 09:04:31.362443 17332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:04:31.362235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42887 (local address 127.0.98.2:53221)
0619 09:04:31.362367 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0619 09:04:31.362404 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":69,"threads_started":1}
I20260619 09:04:31.501284 17333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:04:31.501106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42887 (local address 127.0.98.3:52561)
0619 09:04:31.501211 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0619 09:04:31.501244 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":57,"threads_started":1}
I20260619 09:04:31.559511 17334 raft_consensus.cc:482] T fc2a7f2001204f8eb75ce336ca95afa9 P c9252303f6c147b99263a5c7666a6134 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 09:04:31.559603 17334 raft_consensus.cc:504] T fc2a7f2001204f8eb75ce336ca95afa9 P c9252303f6c147b99263a5c7666a6134 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce39db3fe58a4151a4795ecaf0942bc9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46739 } } peers { permanent_uuid: "51ec1359afd84648a293aff68cab2a8e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33867 } } peers { permanent_uuid: "c9252303f6c147b99263a5c7666a6134" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41245 } }
I20260619 09:04:31.559744 17334 leader_election.cc:288] T fc2a7f2001204f8eb75ce336ca95afa9 P c9252303f6c147b99263a5c7666a6134 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ce39db3fe58a4151a4795ecaf0942bc9 (127.0.98.3:46739), 51ec1359afd84648a293aff68cab2a8e (127.0.98.2:33867)
W20260619 09:04:31.560099 16940 leader_election.cc:341] T fc2a7f2001204f8eb75ce336ca95afa9 P c9252303f6c147b99263a5c7666a6134 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 51ec1359afd84648a293aff68cab2a8e (127.0.98.2:33867): Not found: Tablet not found: fc2a7f2001204f8eb75ce336ca95afa9
W20260619 09:04:31.560181 16937 leader_election.cc:341] T fc2a7f2001204f8eb75ce336ca95afa9 P c9252303f6c147b99263a5c7666a6134 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ce39db3fe58a4151a4795ecaf0942bc9 (127.0.98.3:46739): Not found: Tablet not found: fc2a7f2001204f8eb75ce336ca95afa9
I20260619 09:04:31.560231 16937 leader_election.cc:302] T fc2a7f2001204f8eb75ce336ca95afa9 P c9252303f6c147b99263a5c7666a6134 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c9252303f6c147b99263a5c7666a6134; no voters: 51ec1359afd84648a293aff68cab2a8e, ce39db3fe58a4151a4795ecaf0942bc9
I20260619 09:04:31.560300 17334 raft_consensus.cc:2726] T fc2a7f2001204f8eb75ce336ca95afa9 P c9252303f6c147b99263a5c7666a6134 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 09:04:32.291437 17335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:04:32.291189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42887 (local address 127.0.98.1:37127)
0619 09:04:32.291354 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0619 09:04:32.291390 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":78,"threads_started":1}
I20260619 09:04:32.363183 17336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:04:32.362949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42887 (local address 127.0.98.2:39951)
0619 09:04:32.363101 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0619 09:04:32.363142 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":66,"threads_started":1}
I20260619 09:04:32.501912 17337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:04:32.501702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42887 (local address 127.0.98.3:42229)
0619 09:04:32.501837 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0619 09:04:32.501873 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":57,"threads_started":1}
I20260619 09:04:33.292160 17338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:04:33.291932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42887 (local address 127.0.98.1:55723)
0619 09:04:33.292068 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0619 09:04:33.292109 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":61,"threads_started":1}
I20260619 09:04:33.363835 17339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:04:33.363639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42887 (local address 127.0.98.2:39511)
0619 09:04:33.363750 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0619 09:04:33.363797 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":61,"threads_started":1}
I20260619 09:04:33.502514 17340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:04:33.502329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42887 (local address 127.0.98.3:45487)
0619 09:04:33.502432 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0619 09:04:33.502465 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89,"thread_start_us":54,"threads_started":1}
I20260619 09:04:34.292861 17341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:04:34.292618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42887 (local address 127.0.98.1:56673)
0619 09:04:34.292764 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0619 09:04:34.292814 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":61,"threads_started":1}
I20260619 09:04:34.364493 17342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:04:34.364290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42887 (local address 127.0.98.2:39275)
0619 09:04:34.364409 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0619 09:04:34.364446 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":62,"threads_started":1}
I20260619 09:04:34.503113 17343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:04:34.502927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42887 (local address 127.0.98.3:49417)
0619 09:04:34.503044 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0619 09:04:34.503076 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":65,"threads_started":1}
I20260619 09:04:34.766057 17344 raft_consensus.cc:482] T fc2a7f2001204f8eb75ce336ca95afa9 P c9252303f6c147b99263a5c7666a6134 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 09:04:34.766146 17344 raft_consensus.cc:504] T fc2a7f2001204f8eb75ce336ca95afa9 P c9252303f6c147b99263a5c7666a6134 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce39db3fe58a4151a4795ecaf0942bc9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46739 } } peers { permanent_uuid: "51ec1359afd84648a293aff68cab2a8e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33867 } } peers { permanent_uuid: "c9252303f6c147b99263a5c7666a6134" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41245 } }
I20260619 09:04:34.766316 17344 leader_election.cc:288] T fc2a7f2001204f8eb75ce336ca95afa9 P c9252303f6c147b99263a5c7666a6134 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ce39db3fe58a4151a4795ecaf0942bc9 (127.0.98.3:46739), 51ec1359afd84648a293aff68cab2a8e (127.0.98.2:33867)
W20260619 09:04:34.766630 16940 leader_election.cc:341] T fc2a7f2001204f8eb75ce336ca95afa9 P c9252303f6c147b99263a5c7666a6134 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 51ec1359afd84648a293aff68cab2a8e (127.0.98.2:33867): Not found: Tablet not found: fc2a7f2001204f8eb75ce336ca95afa9
W20260619 09:04:34.766743 16937 leader_election.cc:341] T fc2a7f2001204f8eb75ce336ca95afa9 P c9252303f6c147b99263a5c7666a6134 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ce39db3fe58a4151a4795ecaf0942bc9 (127.0.98.3:46739): Not found: Tablet not found: fc2a7f2001204f8eb75ce336ca95afa9
I20260619 09:04:34.766832 16937 leader_election.cc:302] T fc2a7f2001204f8eb75ce336ca95afa9 P c9252303f6c147b99263a5c7666a6134 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c9252303f6c147b99263a5c7666a6134; no voters: 51ec1359afd84648a293aff68cab2a8e, ce39db3fe58a4151a4795ecaf0942bc9
I20260619 09:04:34.766906 17344 raft_consensus.cc:2726] T fc2a7f2001204f8eb75ce336ca95afa9 P c9252303f6c147b99263a5c7666a6134 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 09:04:35.293603 17345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:04:35.293355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42887 (local address 127.0.98.1:37035)
0619 09:04:35.293507 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0619 09:04:35.293549 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":64,"threads_started":1}
I20260619 09:04:35.365234 17346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:04:35.364977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42887 (local address 127.0.98.2:47393)
0619 09:04:35.365125 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0619 09:04:35.365165 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":73,"threads_started":1}
I20260619 09:04:35.503775 17347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:04:35.503551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42887 (local address 127.0.98.3:49819)
0619 09:04:35.503700 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0619 09:04:35.503733 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":86,"threads_started":1}
I20260619 09:04:36.294332 17348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:04:36.294104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42887 (local address 127.0.98.1:53149)
0619 09:04:36.294250 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0619 09:04:36.294287 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":66,"threads_started":1}
I20260619 09:04:36.365876 17349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:04:36.365670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42887 (local address 127.0.98.2:51269)
0619 09:04:36.365793 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0619 09:04:36.365833 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":66,"threads_started":1}
I20260619 09:04:36.504356 17350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:04:36.504170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42887 (local address 127.0.98.3:34985)
0619 09:04:36.504276 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0619 09:04:36.504315 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89,"thread_start_us":53,"threads_started":1}
I20260619 09:04:36.592197 17351 raft_consensus.cc:482] T fc2a7f2001204f8eb75ce336ca95afa9 P c9252303f6c147b99263a5c7666a6134 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 09:04:36.592298 17351 raft_consensus.cc:504] T fc2a7f2001204f8eb75ce336ca95afa9 P c9252303f6c147b99263a5c7666a6134 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce39db3fe58a4151a4795ecaf0942bc9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46739 } } peers { permanent_uuid: "51ec1359afd84648a293aff68cab2a8e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33867 } } peers { permanent_uuid: "c9252303f6c147b99263a5c7666a6134" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41245 } }
I20260619 09:04:36.592432 17351 leader_election.cc:288] T fc2a7f2001204f8eb75ce336ca95afa9 P c9252303f6c147b99263a5c7666a6134 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ce39db3fe58a4151a4795ecaf0942bc9 (127.0.98.3:46739), 51ec1359afd84648a293aff68cab2a8e (127.0.98.2:33867)
W20260619 09:04:36.592815 16940 leader_election.cc:341] T fc2a7f2001204f8eb75ce336ca95afa9 P c9252303f6c147b99263a5c7666a6134 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 51ec1359afd84648a293aff68cab2a8e (127.0.98.2:33867): Not found: Tablet not found: fc2a7f2001204f8eb75ce336ca95afa9
W20260619 09:04:36.592934 16937 leader_election.cc:341] T fc2a7f2001204f8eb75ce336ca95afa9 P c9252303f6c147b99263a5c7666a6134 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ce39db3fe58a4151a4795ecaf0942bc9 (127.0.98.3:46739): Not found: Tablet not found: fc2a7f2001204f8eb75ce336ca95afa9
I20260619 09:04:36.593015 16937 leader_election.cc:302] T fc2a7f2001204f8eb75ce336ca95afa9 P c9252303f6c147b99263a5c7666a6134 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c9252303f6c147b99263a5c7666a6134; no voters: 51ec1359afd84648a293aff68cab2a8e, ce39db3fe58a4151a4795ecaf0942bc9
I20260619 09:04:36.593093 17351 raft_consensus.cc:2726] T fc2a7f2001204f8eb75ce336ca95afa9 P c9252303f6c147b99263a5c7666a6134 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 09:04:37.295004 17352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:04:37.294748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42887 (local address 127.0.98.1:53255)
0619 09:04:37.294912 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0619 09:04:37.294953 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":110,"threads_started":1}
I20260619 09:04:37.366542 17353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:04:37.366324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42887 (local address 127.0.98.2:43241)
0619 09:04:37.366460 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0619 09:04:37.366499 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":59,"threads_started":1}
I20260619 09:04:37.504946 17354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:04:37.504744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42887 (local address 127.0.98.3:40949)
0619 09:04:37.504861 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0619 09:04:37.504895 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":61,"threads_started":1}
I20260619 09:04:38.295812 17355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:04:38.295517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42887 (local address 127.0.98.1:36967)
0619 09:04:38.295664 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0619 09:04:38.295747 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":65,"threads_started":1}
I20260619 09:04:38.367189 17356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:04:38.366964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42887 (local address 127.0.98.2:43303)
0619 09:04:38.367107 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0619 09:04:38.367146 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":67,"threads_started":1}
I20260619 09:04:38.505571 17357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:04:38.505356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42887 (local address 127.0.98.3:57361)
0619 09:04:38.505491 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0619 09:04:38.505531 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":72,"threads_started":1}
I20260619 09:04:39.296490 17358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:04:39.296279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42887 (local address 127.0.98.1:36231)
0619 09:04:39.296406 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0619 09:04:39.296444 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":58,"threads_started":1}
I20260619 09:04:39.367749 17359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:04:39.367546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42887 (local address 127.0.98.2:59851)
0619 09:04:39.367666 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0619 09:04:39.367700 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":57,"threads_started":1}
I20260619 09:04:39.506242 17360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:04:39.506010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42887 (local address 127.0.98.3:41717)
0619 09:04:39.506153 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0619 09:04:39.506192 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":67,"threads_started":1}
I20260619 09:04:40.234563 17361 raft_consensus.cc:482] T fc2a7f2001204f8eb75ce336ca95afa9 P c9252303f6c147b99263a5c7666a6134 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 09:04:40.234681 17361 raft_consensus.cc:504] T fc2a7f2001204f8eb75ce336ca95afa9 P c9252303f6c147b99263a5c7666a6134 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce39db3fe58a4151a4795ecaf0942bc9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46739 } } peers { permanent_uuid: "51ec1359afd84648a293aff68cab2a8e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33867 } } peers { permanent_uuid: "c9252303f6c147b99263a5c7666a6134" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41245 } }
I20260619 09:04:40.234882 17361 leader_election.cc:288] T fc2a7f2001204f8eb75ce336ca95afa9 P c9252303f6c147b99263a5c7666a6134 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ce39db3fe58a4151a4795ecaf0942bc9 (127.0.98.3:46739), 51ec1359afd84648a293aff68cab2a8e (127.0.98.2:33867)
W20260619 09:04:40.235284 16940 leader_election.cc:341] T fc2a7f2001204f8eb75ce336ca95afa9 P c9252303f6c147b99263a5c7666a6134 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 51ec1359afd84648a293aff68cab2a8e (127.0.98.2:33867): Not found: Tablet not found: fc2a7f2001204f8eb75ce336ca95afa9
W20260619 09:04:40.235370 16937 leader_election.cc:341] T fc2a7f2001204f8eb75ce336ca95afa9 P c9252303f6c147b99263a5c7666a6134 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ce39db3fe58a4151a4795ecaf0942bc9 (127.0.98.3:46739): Not found: Tablet not found: fc2a7f2001204f8eb75ce336ca95afa9
I20260619 09:04:40.235412 16937 leader_election.cc:302] T fc2a7f2001204f8eb75ce336ca95afa9 P c9252303f6c147b99263a5c7666a6134 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c9252303f6c147b99263a5c7666a6134; no voters: 51ec1359afd84648a293aff68cab2a8e, ce39db3fe58a4151a4795ecaf0942bc9
I20260619 09:04:40.235499 17361 raft_consensus.cc:2726] T fc2a7f2001204f8eb75ce336ca95afa9 P c9252303f6c147b99263a5c7666a6134 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 09:04:40.297230 17362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:04:40.296995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42887 (local address 127.0.98.1:41547)
0619 09:04:40.297143 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0619 09:04:40.297184 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":66,"threads_started":1}
I20260619 09:04:40.368358 17363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:04:40.368167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42887 (local address 127.0.98.2:39669)
0619 09:04:40.368278 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0619 09:04:40.368314 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":59,"threads_started":1}
I20260619 09:04:40.506934 17364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:04:40.506659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42887 (local address 127.0.98.3:48675)
0619 09:04:40.506806 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0619 09:04:40.506884 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":60,"threads_started":1}
I20260619 09:04:41.298131 17365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:04:41.297843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42887 (local address 127.0.98.1:54055)
0619 09:04:41.298016 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0619 09:04:41.298063 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":85,"threads_started":1}
I20260619 09:04:41.369185 17366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:04:41.368920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42887 (local address 127.0.98.2:54017)
0619 09:04:41.369090 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0619 09:04:41.369138 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":70,"threads_started":1}
I20260619 09:04:41.507601 17367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:04:41.507365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42887 (local address 127.0.98.3:33087)
0619 09:04:41.507523 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0619 09:04:41.507558 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":87,"threads_started":1}
I20260619 09:04:42.298852 17368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:04:42.298622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42887 (local address 127.0.98.1:44735)
0619 09:04:42.298747 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0619 09:04:42.298783 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":78,"threads_started":1}
I20260619 09:04:42.369899 17369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:04:42.369673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42887 (local address 127.0.98.2:58987)
0619 09:04:42.369808 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0619 09:04:42.369852 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":62,"threads_started":1}
I20260619 09:04:42.508345 17370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:04:42.508086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42887 (local address 127.0.98.3:40231)
0619 09:04:42.508246 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0619 09:04:42.508294 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":73,"threads_started":1}
I20260619 09:04:43.299604 17371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:04:43.299393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42887 (local address 127.0.98.1:55209)
0619 09:04:43.299526 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0619 09:04:43.299558 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":60,"threads_started":1}
I20260619 09:04:43.370592 17372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:04:43.370380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42887 (local address 127.0.98.2:60607)
0619 09:04:43.370511 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0619 09:04:43.370550 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":57,"threads_started":1}
I20260619 09:04:43.508587 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ez5NB/build/release/bin/kudu with pid 16887
I20260619 09:04:43.509330 17373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:04:43.508857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42887 (local address 127.0.98.3:39489)
0619 09:04:43.509249 (+ 392us) negotiation.cc:107] Waiting for socket to connect
0619 09:04:43.509282 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":71,"threads_started":1}
I20260619 09:04:43.512823 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ez5NB/build/release/bin/kudu with pid 17055
I20260619 09:04:43.516417 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ez5NB/build/release/bin/kudu with pid 17188
2026-06-19T09:04:43Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (58283 ms)
[----------] 30 tests from SecurityITest (225387 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781859658134510-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 09:04:43 dist-test-slave-mtdr krb5kdc[17376](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 09:04:43 dist-test-slave-mtdr krb5kdc[17376](info): set up 2 sockets
Jun 19 09:04:43 dist-test-slave-mtdr krb5kdc[17376](info): commencing operation
krb5kdc: starting...
W20260619 09:04:44.474292 392 mini_kdc.cc:121] Time spent starting KDC: real 0.940s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781859658134510-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781859658134510-392-0/krb5kdc/test-user.keytab.
Jun 19 09:04:44 dist-test-slave-mtdr krb5kdc[17376](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859884, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T09:04:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T09:04:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781859658134510-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781859658134510-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 09:04:44.585227 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ez5NB/build/release/bin/kudu
/tmp/dist-test-task4ez5NB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781859658134510-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781859658134510-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781859658134510-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781859658134510-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44537
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781859658134510-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38077
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44537
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781859658134510-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781859658134510-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781859658134510-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 09:04:44.664914 17389 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 09:04:44.665119 17389 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 09:04:44.665141 17389 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 09:04:44.666225 17389 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 09:04:44.666260 17389 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 09:04:44.666273 17389 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 09:04:44.666285 17389 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 09:04:44.666296 17389 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 09:04:44.667697 17389 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38077
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781859658134510-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781859658134510-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44537
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781859658134510-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44537
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781859658134510-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17389
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781859658134510-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 09:04:44.667889 17389 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 09:04:44.668056 17389 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 09:04:44.672648 17394 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:04:44.672652 17397 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:04:44.672832 17395 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:04:44.672905 17389 server_base.cc:1034] running on GCE node
I20260619 09:04:44.673107 17389 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 09:04:44.673364 17389 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 09:04:44.674505 17389 hybrid_clock.cc:648] HybridClock initialized: now 1781859884674491 us; error 28 us; skew 500 ppm
Jun 19 09:04:44 dist-test-slave-mtdr krb5kdc[17376](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859884, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 09:04:44.676899 17389 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 09:04:44.677291 17389 webserver.cc:460] Webserver started at http://127.0.98.62:41887/ using document root <none> and password file <none>
I20260619 09:04:44.677457 17389 fs_manager.cc:359] Metadata directory not provided
I20260619 09:04:44.677495 17389 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 09:04:44.677583 17389 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 09:04:44.678556 17389 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781859658134510-392-0/minicluster-data/master-0/data/instance:
uuid: "7359ae3596f043f1980850074eb576e8"
format_stamp: "Formatted at 2026-06-19 09:04:44 on dist-test-slave-mtdr"
server_key: "43bf2375d8060fc3a77a69011caee83c"
server_key_iv: "64a17ae460a5ae1e2c6a4b027e38857b"
server_key_version: "encryptionkey@0"
I20260619 09:04:44.678882 17389 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781859658134510-392-0/minicluster-data/master-0/wal/instance:
uuid: "7359ae3596f043f1980850074eb576e8"
format_stamp: "Formatted at 2026-06-19 09:04:44 on dist-test-slave-mtdr"
server_key: "43bf2375d8060fc3a77a69011caee83c"
server_key_iv: "64a17ae460a5ae1e2c6a4b027e38857b"
server_key_version: "encryptionkey@0"
I20260619 09:04:44.680023 17389 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 09:04:44.680625 17404 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 09:04:44.680764 17389 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 09:04:44.680840 17389 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781859658134510-392-0/minicluster-data/master-0/data,/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781859658134510-392-0/minicluster-data/master-0/wal
uuid: "7359ae3596f043f1980850074eb576e8"
format_stamp: "Formatted at 2026-06-19 09:04:44 on dist-test-slave-mtdr"
server_key: "43bf2375d8060fc3a77a69011caee83c"
server_key_iv: "64a17ae460a5ae1e2c6a4b027e38857b"
server_key_version: "encryptionkey@0"
I20260619 09:04:44.680903 17389 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781859658134510-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781859658134510-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781859658134510-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 09:04:44.694989 17389 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 09:04:44.697000 17389 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 09:04:44.697113 17389 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 09:04:44.701064 17389 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44537
I20260619 09:04:44.701167 17456 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44537 every 8 connection(s)
I20260619 09:04:44.701403 17389 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781859658134510-392-0/minicluster-data/master-0/data/info.pb
I20260619 09:04:44.702169 17457 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 09:04:44.704396 17457 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7359ae3596f043f1980850074eb576e8: Bootstrap starting.
I20260619 09:04:44.704914 17457 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7359ae3596f043f1980850074eb576e8: Neither blocks nor log segments found. Creating new log.
I20260619 09:04:44.705156 17457 log.cc:824] T 00000000000000000000000000000000 P 7359ae3596f043f1980850074eb576e8: Log is configured to *not* fsync() on all Append() calls
I20260619 09:04:44.705750 17457 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7359ae3596f043f1980850074eb576e8: No bootstrap required, opened a new log
I20260619 09:04:44.707305 17457 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7359ae3596f043f1980850074eb576e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7359ae3596f043f1980850074eb576e8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44537 } }
I20260619 09:04:44.707433 17457 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7359ae3596f043f1980850074eb576e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 09:04:44.707463 17457 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7359ae3596f043f1980850074eb576e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7359ae3596f043f1980850074eb576e8, State: Initialized, Role: FOLLOWER
I20260619 09:04:44.707638 17457 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7359ae3596f043f1980850074eb576e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7359ae3596f043f1980850074eb576e8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44537 } }
I20260619 09:04:44.707760 17457 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7359ae3596f043f1980850074eb576e8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 09:04:44.707796 17457 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7359ae3596f043f1980850074eb576e8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 09:04:44.707844 17457 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7359ae3596f043f1980850074eb576e8 [term 0 FOLLOWER]: Advancing to term 1
I20260619 09:04:44.708323 17457 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7359ae3596f043f1980850074eb576e8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7359ae3596f043f1980850074eb576e8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44537 } }
I20260619 09:04:44.708413 17457 leader_election.cc:302] T 00000000000000000000000000000000 P 7359ae3596f043f1980850074eb576e8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7359ae3596f043f1980850074eb576e8; no voters:
I20260619 09:04:44.708554 17457 leader_election.cc:288] T 00000000000000000000000000000000 P 7359ae3596f043f1980850074eb576e8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 09:04:44.708591 17460 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7359ae3596f043f1980850074eb576e8 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 09:04:44.708739 17460 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7359ae3596f043f1980850074eb576e8 [term 1 LEADER]: Becoming Leader. State: Replica: 7359ae3596f043f1980850074eb576e8, State: Running, Role: LEADER
I20260619 09:04:44.708765 17457 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7359ae3596f043f1980850074eb576e8 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 09:04:44.708827 17460 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7359ae3596f043f1980850074eb576e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7359ae3596f043f1980850074eb576e8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44537 } }
I20260619 09:04:44.709132 17460 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7359ae3596f043f1980850074eb576e8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7359ae3596f043f1980850074eb576e8. Latest consensus state: current_term: 1 leader_uuid: "7359ae3596f043f1980850074eb576e8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7359ae3596f043f1980850074eb576e8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44537 } } }
I20260619 09:04:44.709244 17460 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7359ae3596f043f1980850074eb576e8 [sys.catalog]: This master's current role is: LEADER
I20260619 09:04:44.709339 17462 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7359ae3596f043f1980850074eb576e8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7359ae3596f043f1980850074eb576e8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7359ae3596f043f1980850074eb576e8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44537 } } }
I20260619 09:04:44.709656 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ez5NB/build/release/bin/kudu as pid 17389
I20260619 09:04:44.709781 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781859658134510-392-0/minicluster-data/master-0/wal/instance
I20260619 09:04:44.709981 392 external_mini_cluster.cc:1428] Setting key 6995095ff22c25e98d50432b3684c216
I20260619 09:04:44.709653 17469 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 09:04:44.710101 17462 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7359ae3596f043f1980850074eb576e8 [sys.catalog]: This master's current role is: LEADER
I20260619 09:04:44.710556 17469 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 09:04:44.712858 17469 catalog_manager.cc:1347] Generated new cluster ID: b49351f092fb4c1a83c80106259284ee
I20260619 09:04:44.712909 17469 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 19 09:04:44 dist-test-slave-mtdr krb5kdc[17376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859884, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 09:04:44.720373 17479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:04:44.710411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44004 (local address 127.0.98.62:44537)
0619 09:04:44.710643 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0619 09:04:44.710652 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 09:04:44.710680 (+ 28us) server_negotiation.cc:407] Connection header received
0619 09:04:44.710799 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:04:44.710806 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:04:44.710940 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:04:44.711100 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:04:44.711821 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:04:44.712726 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:04:44.713727 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:04:44.714368 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:04:44.714481 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:04:44.717075 (+ 2594us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:04:44.717078 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:04:44.717079 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:04:44.717090 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:04:44.718885 (+ 1795us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:04:44.719240 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:04:44.719242 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:04:44.719243 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:04:44.719339 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:04:44.719514 (+ 175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:04:44.719515 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:04:44.719515 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:04:44.719787 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:04:44.719921 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0619 09:04:44.720027 (+ 106us) server_negotiation.cc:299] Negotiation successful
0619 09:04:44.720150 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":188,"thread_start_us":80,"threads_started":1}
I20260619 09:04:44.722157 17469 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 09:04:44.722596 17469 catalog_manager.cc:1504] Loading token signing keys...
I20260619 09:04:44.725925 17469 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7359ae3596f043f1980850074eb576e8: Generated new TSK 0
I20260619 09:04:44.726070 17469 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781859658134510-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781859658134510-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 09:04:44.753054 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ez5NB/build/release/bin/kudu
/tmp/dist-test-task4ez5NB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781859658134510-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781859658134510-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781859658134510-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781859658134510-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44537
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781859658134510-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38077
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781859658134510-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781859658134510-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781859658134510-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 09:04:44.832114 17483 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 09:04:44.832304 17483 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 09:04:44.832325 17483 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 09:04:44.833432 17483 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 09:04:44.833472 17483 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 09:04:44.833495 17483 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 09:04:44.834954 17483 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38077
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781859658134510-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781859658134510-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781859658134510-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781859658134510-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44537
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17483
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781859658134510-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 09:04:44.835155 17483 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 09:04:44.835351 17483 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 09:04:44.839886 17489 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:04:44.839902 17491 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:04:44.839882 17488 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:04:44.839939 17483 server_base.cc:1034] running on GCE node
I20260619 09:04:44.840286 17483 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 09:04:44.840502 17483 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 09:04:44.841639 17483 hybrid_clock.cc:648] HybridClock initialized: now 1781859884841626 us; error 31 us; skew 500 ppm
Jun 19 09:04:44 dist-test-slave-mtdr krb5kdc[17376](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859884, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 09:04:44.844053 17483 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 09:04:44.844440 17483 webserver.cc:460] Webserver started at http://127.0.98.1:45395/ using document root <none> and password file <none>
I20260619 09:04:44.844645 17483 fs_manager.cc:359] Metadata directory not provided
I20260619 09:04:44.844693 17483 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 09:04:44.844811 17483 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 09:04:44.845764 17483 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781859658134510-392-0/minicluster-data/ts-0/data/instance:
uuid: "5920513c169740c1905c47c275c337fa"
format_stamp: "Formatted at 2026-06-19 09:04:44 on dist-test-slave-mtdr"
server_key: "9aa77e264791aa30d1b998c7fc414846"
server_key_iv: "0d6b1543d3a58d46efe59fd9b669d3ec"
server_key_version: "encryptionkey@0"
I20260619 09:04:44.846072 17483 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781859658134510-392-0/minicluster-data/ts-0/wal/instance:
uuid: "5920513c169740c1905c47c275c337fa"
format_stamp: "Formatted at 2026-06-19 09:04:44 on dist-test-slave-mtdr"
server_key: "9aa77e264791aa30d1b998c7fc414846"
server_key_iv: "0d6b1543d3a58d46efe59fd9b669d3ec"
server_key_version: "encryptionkey@0"
I20260619 09:04:44.847325 17483 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 09:04:44.848001 17498 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 09:04:44.848163 17483 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 09:04:44.848240 17483 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781859658134510-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781859658134510-392-0/minicluster-data/ts-0/wal
uuid: "5920513c169740c1905c47c275c337fa"
format_stamp: "Formatted at 2026-06-19 09:04:44 on dist-test-slave-mtdr"
server_key: "9aa77e264791aa30d1b998c7fc414846"
server_key_iv: "0d6b1543d3a58d46efe59fd9b669d3ec"
server_key_version: "encryptionkey@0"
I20260619 09:04:44.848297 17483 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781859658134510-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781859658134510-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781859658134510-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 09:04:44.857800 17483 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 09:04:44.859632 17483 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 09:04:44.859757 17483 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 09:04:44.859941 17483 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 09:04:44.860209 17483 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 09:04:44.860244 17483 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 09:04:44.860270 17483 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 09:04:44.860288 17483 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 09:04:44.865509 17483 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38613
I20260619 09:04:44.865548 17611 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38613 every 8 connection(s)
I20260619 09:04:44.865832 17483 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781859658134510-392-0/minicluster-data/ts-0/data/info.pb
I20260619 09:04:44.867136 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ez5NB/build/release/bin/kudu as pid 17483
I20260619 09:04:44.867233 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781859658134510-392-0/minicluster-data/ts-0/wal/instance
I20260619 09:04:44.867414 392 external_mini_cluster.cc:1428] Setting key b08d540c6dbb801afb93b2edd66b626c
Jun 19 09:04:44 dist-test-slave-mtdr krb5kdc[17376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859884, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260619 09:04:44.875551 17479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:04:44.866507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44813 (local address 127.0.98.62:44537)
0619 09:04:44.866579 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0619 09:04:44.866581 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:04:44.866725 (+ 144us) server_negotiation.cc:407] Connection header received
0619 09:04:44.867698 (+ 973us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:04:44.867699 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:04:44.867715 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:04:44.867767 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:04:44.868383 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:04:44.869076 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:04:44.870239 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:04:44.870431 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:04:44.870482 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:04:44.872828 (+ 2346us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:04:44.872829 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:04:44.872829 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:04:44.872836 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:04:44.874086 (+ 1250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:04:44.874520 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:04:44.874521 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:04:44.874521 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:04:44.874573 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:04:44.874933 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:04:44.874934 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:04:44.874934 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:04:44.875046 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:04:44.875115 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0619 09:04:44.875417 (+ 302us) server_negotiation.cc:299] Negotiation successful
0619 09:04:44.875486 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260619 09:04:44.875674 17614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:04:44.866473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44537 (local address 127.0.98.1:44813)
0619 09:04:44.866615 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0619 09:04:44.866649 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0619 09:04:44.867606 (+ 957us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:04:44.867823 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:04:44.867826 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:04:44.868113 (+ 287us) client_negotiation.cc:189] Negotiated authn=SASL
0619 09:04:44.868305 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:04:44.868310 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:04:44.869184 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:04:44.869185 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:04:44.870156 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:04:44.870159 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:04:44.870504 (+ 345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:04:44.870505 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:04:44.870639 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:04:44.871161 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 09:04:44.871171 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 09:04:44.872722 (+ 1551us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 09:04:44.874166 (+ 1444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:04:44.874172 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:04:44.874174 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:04:44.874445 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:04:44.874657 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:04:44.874659 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:04:44.874659 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:04:44.874794 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:04:44.875117 (+ 323us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 09:04:44.875124 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 09:04:44.875282 (+ 158us) client_negotiation.cc:769] Sending connection context
0619 09:04:44.875386 (+ 104us) client_negotiation.cc:240] Negotiation successful
0619 09:04:44.875506 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":69,"threads_started":1}
I20260619 09:04:44.876122 17612 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44537
I20260619 09:04:44.876204 17612 heartbeater.cc:464] Registering TS with master...
I20260619 09:04:44.876344 17612 heartbeater.cc:511] Master 127.0.98.62:44537 requested a full tablet report, sending...
I20260619 09:04:44.876796 17421 ts_manager.cc:194] Registered new tserver with Master: 5920513c169740c1905c47c275c337fa (127.0.98.1:38613)
I20260619 09:04:44.877432 17421 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:44813
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781859658134510-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781859658134510-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 09:04:44.892052 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ez5NB/build/release/bin/kudu
/tmp/dist-test-task4ez5NB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781859658134510-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781859658134510-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781859658134510-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781859658134510-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44537
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781859658134510-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38077
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781859658134510-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781859658134510-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781859658134510-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 09:04:44.968534 17617 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 09:04:44.968726 17617 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 09:04:44.968756 17617 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 09:04:44.969890 17617 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 09:04:44.969929 17617 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 09:04:44.969964 17617 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 09:04:44.971453 17617 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38077
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781859658134510-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781859658134510-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781859658134510-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781859658134510-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44537
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17617
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781859658134510-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 09:04:44.971676 17617 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 09:04:44.971868 17617 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 09:04:44.976459 17625 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:04:44.976434 17622 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:04:44.976482 17617 server_base.cc:1034] running on GCE node
W20260619 09:04:44.976434 17623 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:04:44.976883 17617 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 09:04:44.977098 17617 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 09:04:44.978250 17617 hybrid_clock.cc:648] HybridClock initialized: now 1781859884978238 us; error 28 us; skew 500 ppm
Jun 19 09:04:44 dist-test-slave-mtdr krb5kdc[17376](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859884, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 09:04:44.980681 17617 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 09:04:44.981094 17617 webserver.cc:460] Webserver started at http://127.0.98.2:46183/ using document root <none> and password file <none>
I20260619 09:04:44.981293 17617 fs_manager.cc:359] Metadata directory not provided
I20260619 09:04:44.981339 17617 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 09:04:44.981453 17617 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 09:04:44.982437 17617 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781859658134510-392-0/minicluster-data/ts-1/data/instance:
uuid: "9880f9b2fbcd4a509532112267e1d25e"
format_stamp: "Formatted at 2026-06-19 09:04:44 on dist-test-slave-mtdr"
server_key: "6755582ed062d96320734b5cdd9ae743"
server_key_iv: "745a08bb54c9f3ce23feb9cebe4085ec"
server_key_version: "encryptionkey@0"
I20260619 09:04:44.982750 17617 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781859658134510-392-0/minicluster-data/ts-1/wal/instance:
uuid: "9880f9b2fbcd4a509532112267e1d25e"
format_stamp: "Formatted at 2026-06-19 09:04:44 on dist-test-slave-mtdr"
server_key: "6755582ed062d96320734b5cdd9ae743"
server_key_iv: "745a08bb54c9f3ce23feb9cebe4085ec"
server_key_version: "encryptionkey@0"
I20260619 09:04:44.984012 17617 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260619 09:04:44.984757 17632 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 09:04:44.984908 17617 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260619 09:04:44.984984 17617 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781859658134510-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781859658134510-392-0/minicluster-data/ts-1/wal
uuid: "9880f9b2fbcd4a509532112267e1d25e"
format_stamp: "Formatted at 2026-06-19 09:04:44 on dist-test-slave-mtdr"
server_key: "6755582ed062d96320734b5cdd9ae743"
server_key_iv: "745a08bb54c9f3ce23feb9cebe4085ec"
server_key_version: "encryptionkey@0"
I20260619 09:04:44.985034 17617 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781859658134510-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781859658134510-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781859658134510-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 09:04:44.994807 17617 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 09:04:44.996536 17617 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 09:04:44.996640 17617 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 09:04:44.996795 17617 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 09:04:44.997016 17617 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 09:04:44.997044 17617 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 09:04:44.997063 17617 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 09:04:44.997082 17617 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 09:04:45.002246 17617 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41383
I20260619 09:04:45.002303 17745 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41383 every 8 connection(s)
I20260619 09:04:45.002556 17617 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781859658134510-392-0/minicluster-data/ts-1/data/info.pb
I20260619 09:04:45.006371 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ez5NB/build/release/bin/kudu as pid 17617
I20260619 09:04:45.006453 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781859658134510-392-0/minicluster-data/ts-1/wal/instance
I20260619 09:04:45.006635 392 external_mini_cluster.cc:1428] Setting key 4d7f7204fa48f3490a596176f7b0cd69
Jun 19 09:04:45 dist-test-slave-mtdr krb5kdc[17376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859884, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 09:04:45.012528 17479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:04:45.003298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35049 (local address 127.0.98.62:44537)
0619 09:04:45.003371 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0619 09:04:45.003373 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:04:45.003530 (+ 157us) server_negotiation.cc:407] Connection header received
0619 09:04:45.004477 (+ 947us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:04:45.004479 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:04:45.004493 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:04:45.004566 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:04:45.005175 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:04:45.005709 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:04:45.006750 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:04:45.007046 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:04:45.007113 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:04:45.009644 (+ 2531us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:04:45.009645 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:04:45.009645 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:04:45.009652 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:04:45.011272 (+ 1620us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:04:45.011632 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:04:45.011633 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:04:45.011634 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:04:45.011699 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:04:45.011916 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:04:45.011917 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:04:45.011917 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:04:45.012054 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:04:45.012132 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0619 09:04:45.012345 (+ 213us) server_negotiation.cc:299] Negotiation successful
0619 09:04:45.012429 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260619 09:04:45.012563 17748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:04:45.003240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44537 (local address 127.0.98.2:35049)
0619 09:04:45.003408 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0619 09:04:45.003449 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 09:04:45.004355 (+ 906us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:04:45.004641 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:04:45.004645 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:04:45.004888 (+ 243us) client_negotiation.cc:189] Negotiated authn=SASL
0619 09:04:45.005076 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:04:45.005081 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:04:45.005797 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:04:45.005798 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:04:45.006666 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:04:45.006669 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:04:45.007121 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:04:45.007122 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:04:45.007254 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:04:45.007740 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 09:04:45.007751 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 09:04:45.009560 (+ 1809us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 09:04:45.011352 (+ 1792us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:04:45.011358 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:04:45.011360 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:04:45.011559 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:04:45.011765 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:04:45.011766 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:04:45.011766 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:04:45.011858 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:04:45.012117 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 09:04:45.012120 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 09:04:45.012230 (+ 110us) client_negotiation.cc:769] Sending connection context
0619 09:04:45.012304 (+ 74us) client_negotiation.cc:240] Negotiation successful
0619 09:04:45.012414 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":81,"threads_started":1}
I20260619 09:04:45.013068 17746 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44537
I20260619 09:04:45.013171 17746 heartbeater.cc:464] Registering TS with master...
I20260619 09:04:45.013363 17746 heartbeater.cc:511] Master 127.0.98.62:44537 requested a full tablet report, sending...
I20260619 09:04:45.013746 17421 ts_manager.cc:194] Registered new tserver with Master: 9880f9b2fbcd4a509532112267e1d25e (127.0.98.2:41383)
I20260619 09:04:45.014113 17421 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:35049
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781859658134510-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781859658134510-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 09:04:45.031957 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ez5NB/build/release/bin/kudu
/tmp/dist-test-task4ez5NB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781859658134510-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781859658134510-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781859658134510-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781859658134510-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44537
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781859658134510-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38077
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781859658134510-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781859658134510-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781859658134510-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 09:04:45.110954 17751 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 09:04:45.111178 17751 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 09:04:45.111203 17751 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 09:04:45.112385 17751 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 09:04:45.112426 17751 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 09:04:45.112453 17751 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 09:04:45.113936 17751 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38077
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781859658134510-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781859658134510-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781859658134510-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781859658134510-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44537
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17751
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781859658134510-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 09:04:45.114154 17751 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 09:04:45.114377 17751 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 09:04:45.119181 17756 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:04:45.119194 17757 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:04:45.119267 17759 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:04:45.119333 17751 server_base.cc:1034] running on GCE node
I20260619 09:04:45.119621 17751 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 09:04:45.119836 17751 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 09:04:45.120977 17751 hybrid_clock.cc:648] HybridClock initialized: now 1781859885120964 us; error 35 us; skew 500 ppm
Jun 19 09:04:45 dist-test-slave-mtdr krb5kdc[17376](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859885, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 09:04:45.123589 17751 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 09:04:45.124040 17751 webserver.cc:460] Webserver started at http://127.0.98.3:41207/ using document root <none> and password file <none>
I20260619 09:04:45.124238 17751 fs_manager.cc:359] Metadata directory not provided
I20260619 09:04:45.124286 17751 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 09:04:45.124408 17751 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 09:04:45.125430 17751 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781859658134510-392-0/minicluster-data/ts-2/data/instance:
uuid: "cb22cee68ee644ac9571f9ccb2c91190"
format_stamp: "Formatted at 2026-06-19 09:04:45 on dist-test-slave-mtdr"
server_key: "ab08be05a39e001f330996876e91e7ac"
server_key_iv: "aa30e20ef3ebbbf23511935943644a2b"
server_key_version: "encryptionkey@0"
I20260619 09:04:45.125756 17751 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781859658134510-392-0/minicluster-data/ts-2/wal/instance:
uuid: "cb22cee68ee644ac9571f9ccb2c91190"
format_stamp: "Formatted at 2026-06-19 09:04:45 on dist-test-slave-mtdr"
server_key: "ab08be05a39e001f330996876e91e7ac"
server_key_iv: "aa30e20ef3ebbbf23511935943644a2b"
server_key_version: "encryptionkey@0"
I20260619 09:04:45.127033 17751 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 09:04:45.127735 17766 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 09:04:45.127977 17751 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 09:04:45.128069 17751 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781859658134510-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781859658134510-392-0/minicluster-data/ts-2/wal
uuid: "cb22cee68ee644ac9571f9ccb2c91190"
format_stamp: "Formatted at 2026-06-19 09:04:45 on dist-test-slave-mtdr"
server_key: "ab08be05a39e001f330996876e91e7ac"
server_key_iv: "aa30e20ef3ebbbf23511935943644a2b"
server_key_version: "encryptionkey@0"
I20260619 09:04:45.128129 17751 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781859658134510-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781859658134510-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781859658134510-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 09:04:45.152073 17751 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 09:04:45.153954 17751 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 09:04:45.154098 17751 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 09:04:45.154315 17751 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 09:04:45.154614 17751 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 09:04:45.154650 17751 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 09:04:45.154680 17751 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 09:04:45.154697 17751 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 09:04:45.160058 17751 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44461
I20260619 09:04:45.160100 17879 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44461 every 8 connection(s)
I20260619 09:04:45.160394 17751 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781859658134510-392-0/minicluster-data/ts-2/data/info.pb
Jun 19 09:04:45 dist-test-slave-mtdr krb5kdc[17376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859885, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 09:04:45.166491 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ez5NB/build/release/bin/kudu as pid 17751
I20260619 09:04:45.166594 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781859658134510-392-0/minicluster-data/ts-2/wal/instance
I20260619 09:04:45.166803 392 external_mini_cluster.cc:1428] Setting key 8122942f89b42a351923bcad44bbcd86
I20260619 09:04:45.169343 17479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:04:45.161089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43059 (local address 127.0.98.62:44537)
0619 09:04:45.161154 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0619 09:04:45.161156 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:04:45.161342 (+ 186us) server_negotiation.cc:407] Connection header received
0619 09:04:45.162251 (+ 909us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:04:45.162253 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:04:45.162265 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:04:45.162316 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:04:45.162929 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:04:45.163422 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:04:45.164388 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:04:45.164590 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:04:45.164644 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:04:45.166758 (+ 2114us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:04:45.166759 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:04:45.166760 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:04:45.166766 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:04:45.168138 (+ 1372us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:04:45.168480 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:04:45.168481 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:04:45.168482 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:04:45.168539 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:04:45.168744 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:04:45.168745 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:04:45.168745 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:04:45.168853 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:04:45.168906 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0619 09:04:45.169171 (+ 265us) server_negotiation.cc:299] Negotiation successful
0619 09:04:45.169259 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260619 09:04:45.169503 17882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:04:45.161072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44537 (local address 127.0.98.3:43059)
0619 09:04:45.161232 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0619 09:04:45.161265 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0619 09:04:45.162145 (+ 880us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:04:45.162384 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:04:45.162388 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:04:45.162640 (+ 252us) client_negotiation.cc:189] Negotiated authn=SASL
0619 09:04:45.162845 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:04:45.162850 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:04:45.163514 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:04:45.163514 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:04:45.164310 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:04:45.164312 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:04:45.164656 (+ 344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:04:45.164657 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:04:45.164771 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:04:45.165205 (+ 434us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 09:04:45.165213 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 09:04:45.166629 (+ 1416us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 09:04:45.168209 (+ 1580us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:04:45.168216 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:04:45.168218 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:04:45.168418 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:04:45.168590 (+ 172us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:04:45.168591 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:04:45.168591 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:04:45.168691 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:04:45.168906 (+ 215us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 09:04:45.168912 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 09:04:45.169045 (+ 133us) client_negotiation.cc:769] Sending connection context
0619 09:04:45.169128 (+ 83us) client_negotiation.cc:240] Negotiation successful
0619 09:04:45.169292 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":86,"threads_started":1}
I20260619 09:04:45.170029 17880 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44537
I20260619 09:04:45.170128 17880 heartbeater.cc:464] Registering TS with master...
I20260619 09:04:45.170289 17880 heartbeater.cc:511] Master 127.0.98.62:44537 requested a full tablet report, sending...
I20260619 09:04:45.170601 17421 ts_manager.cc:194] Registered new tserver with Master: cb22cee68ee644ac9571f9ccb2c91190 (127.0.98.3:44461)
I20260619 09:04:45.171021 17421 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:43059
I20260619 09:04:45.178399 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 09:04:45.185307 17479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:04:45.179683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44010 (local address 127.0.98.62:44537)
0619 09:04:45.179742 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0619 09:04:45.179744 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:04:45.179853 (+ 109us) server_negotiation.cc:407] Connection header received
0619 09:04:45.179903 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:04:45.179904 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:04:45.179914 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:04:45.179959 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:04:45.180473 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:04:45.180947 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:04:45.181652 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:04:45.181845 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:04:45.181897 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:04:45.183086 (+ 1189us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:04:45.183087 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:04:45.183088 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:04:45.183095 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:04:45.184308 (+ 1213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:04:45.184605 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:04:45.184606 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:04:45.184607 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:04:45.184656 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:04:45.184811 (+ 155us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:04:45.184812 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:04:45.184812 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:04:45.184945 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:04:45.185031 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0619 09:04:45.185153 (+ 122us) server_negotiation.cc:299] Negotiation successful
0619 09:04:45.185223 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260619 09:04:45.186735 17421 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44010:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 09:04:45.188374 17421 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 09:04:45.194728 17890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:04:45.190563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41182 (local address 127.0.98.1:38613)
0619 09:04:45.191129 (+ 566us) server_negotiation.cc:206] Beginning negotiation
0619 09:04:45.191134 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 09:04:45.191144 (+ 10us) server_negotiation.cc:407] Connection header received
0619 09:04:45.191186 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:04:45.191188 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:04:45.191203 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:04:45.191284 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 09:04:45.191424 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:04:45.192517 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:04:45.193605 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:04:45.194286 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:04:45.194378 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:04:45.194442 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0619 09:04:45.194591 (+ 149us) server_negotiation.cc:299] Negotiation successful
0619 09:04:45.194636 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":529,"thread_start_us":112,"threads_started":1}
I20260619 09:04:45.194720 17889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:04:45.190425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38613 (local address 127.0.0.1:41182)
0619 09:04:45.190574 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0619 09:04:45.190613 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 09:04:45.190677 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:04:45.191297 (+ 620us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:04:45.191299 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:04:45.191300 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 09:04:45.191342 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:04:45.191344 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:04:45.192600 (+ 1256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:04:45.192601 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:04:45.193536 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:04:45.193538 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:04:45.194388 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:04:45.194389 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:04:45.194477 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:04:45.194491 (+ 14us) client_negotiation.cc:769] Sending connection context
0619 09:04:45.194570 (+ 79us) client_negotiation.cc:240] Negotiation successful
0619 09:04:45.194626 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"mutex_wait_us":23,"thread_start_us":97,"threads_started":1}
I20260619 09:04:45.194736 17891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:04:45.190655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41383 (local address 127.0.0.1:41278)
0619 09:04:45.191399 (+ 744us) negotiation.cc:107] Waiting for socket to connect
0619 09:04:45.191409 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 09:04:45.191466 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:04:45.191657 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:04:45.191658 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:04:45.191659 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 09:04:45.191709 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:04:45.191712 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:04:45.192660 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:04:45.192661 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:04:45.193775 (+ 1114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:04:45.193777 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:04:45.194527 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:04:45.194529 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:04:45.194582 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:04:45.194585 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 09:04:45.194631 (+ 46us) client_negotiation.cc:240] Negotiation successful
0619 09:04:45.194660 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":726,"thread_start_us":58,"threads_started":1}
I20260619 09:04:45.195103 17888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:04:45.190430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44461 (local address 127.0.0.1:59026)
0619 09:04:45.190761 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0619 09:04:45.190772 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 09:04:45.190842 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:04:45.191017 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:04:45.191021 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:04:45.191023 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 09:04:45.191091 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:04:45.191093 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:04:45.192393 (+ 1300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:04:45.192394 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:04:45.193749 (+ 1355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:04:45.193752 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:04:45.194856 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:04:45.194858 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:04:45.194924 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:04:45.194928 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 09:04:45.194976 (+ 48us) client_negotiation.cc:240] Negotiation successful
0619 09:04:45.195036 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":318,"mutex_wait_us":4,"thread_start_us":65,"threads_started":1}
I20260619 09:04:45.195116 17892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:04:45.190698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59026 (local address 127.0.98.3:44461)
0619 09:04:45.190869 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0619 09:04:45.190874 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 09:04:45.190885 (+ 11us) server_negotiation.cc:407] Connection header received
0619 09:04:45.190917 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:04:45.190918 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:04:45.190930 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:04:45.190995 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 09:04:45.191192 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:04:45.192286 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:04:45.193834 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:04:45.194511 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:04:45.194580 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:04:45.194630 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0619 09:04:45.195002 (+ 372us) server_negotiation.cc:299] Negotiation successful
0619 09:04:45.195042 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":79,"threads_started":1}
I20260619 09:04:45.195631 17546 tablet_service.cc:1467] Processing CreateTablet for tablet d2d679d7e23540fd88c9df59e74f128d (DEFAULT_TABLE table=test-table [id=e60a08fabcc54181a1af0e5ad723550f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 09:04:45.195844 17814 tablet_service.cc:1467] Processing CreateTablet for tablet d2d679d7e23540fd88c9df59e74f128d (DEFAULT_TABLE table=test-table [id=e60a08fabcc54181a1af0e5ad723550f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 09:04:45.196035 17546 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2d679d7e23540fd88c9df59e74f128d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 09:04:45.196169 17814 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2d679d7e23540fd88c9df59e74f128d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 09:04:45.198695 17894 tablet_bootstrap.cc:492] T d2d679d7e23540fd88c9df59e74f128d P cb22cee68ee644ac9571f9ccb2c91190: Bootstrap starting.
I20260619 09:04:45.198763 17895 tablet_bootstrap.cc:492] T d2d679d7e23540fd88c9df59e74f128d P 5920513c169740c1905c47c275c337fa: Bootstrap starting.
I20260619 09:04:45.199045 17893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:04:45.190883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41278 (local address 127.0.98.2:41383)
0619 09:04:45.191087 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0619 09:04:45.191091 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 09:04:45.191528 (+ 437us) server_negotiation.cc:407] Connection header received
0619 09:04:45.191571 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:04:45.191573 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:04:45.191585 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:04:45.191652 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 09:04:45.191780 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:04:45.192538 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:04:45.193866 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:04:45.194361 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:04:45.194434 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:04:45.194492 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0619 09:04:45.198911 (+ 4419us) server_negotiation.cc:299] Negotiation successful
0619 09:04:45.198981 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":108,"threads_started":1}
I20260619 09:04:45.199452 17894 tablet_bootstrap.cc:654] T d2d679d7e23540fd88c9df59e74f128d P cb22cee68ee644ac9571f9ccb2c91190: Neither blocks nor log segments found. Creating new log.
I20260619 09:04:45.199451 17895 tablet_bootstrap.cc:654] T d2d679d7e23540fd88c9df59e74f128d P 5920513c169740c1905c47c275c337fa: Neither blocks nor log segments found. Creating new log.
I20260619 09:04:45.199640 17680 tablet_service.cc:1467] Processing CreateTablet for tablet d2d679d7e23540fd88c9df59e74f128d (DEFAULT_TABLE table=test-table [id=e60a08fabcc54181a1af0e5ad723550f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 09:04:45.199927 17680 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2d679d7e23540fd88c9df59e74f128d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 09:04:45.200024 17894 log.cc:824] T d2d679d7e23540fd88c9df59e74f128d P cb22cee68ee644ac9571f9ccb2c91190: Log is configured to *not* fsync() on all Append() calls
I20260619 09:04:45.200026 17895 log.cc:824] T d2d679d7e23540fd88c9df59e74f128d P 5920513c169740c1905c47c275c337fa: Log is configured to *not* fsync() on all Append() calls
I20260619 09:04:45.200770 17895 tablet_bootstrap.cc:492] T d2d679d7e23540fd88c9df59e74f128d P 5920513c169740c1905c47c275c337fa: No bootstrap required, opened a new log
I20260619 09:04:45.200845 17895 ts_tablet_manager.cc:1397] T d2d679d7e23540fd88c9df59e74f128d P 5920513c169740c1905c47c275c337fa: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260619 09:04:45.201021 17894 tablet_bootstrap.cc:492] T d2d679d7e23540fd88c9df59e74f128d P cb22cee68ee644ac9571f9ccb2c91190: No bootstrap required, opened a new log
I20260619 09:04:45.201087 17894 ts_tablet_manager.cc:1397] T d2d679d7e23540fd88c9df59e74f128d P cb22cee68ee644ac9571f9ccb2c91190: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260619 09:04:45.202890 17898 tablet_bootstrap.cc:492] T d2d679d7e23540fd88c9df59e74f128d P 9880f9b2fbcd4a509532112267e1d25e: Bootstrap starting.
I20260619 09:04:45.202917 17895 raft_consensus.cc:348] T d2d679d7e23540fd88c9df59e74f128d P 5920513c169740c1905c47c275c337fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5920513c169740c1905c47c275c337fa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38613 } } peers { permanent_uuid: "9880f9b2fbcd4a509532112267e1d25e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41383 } } peers { permanent_uuid: "cb22cee68ee644ac9571f9ccb2c91190" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44461 } }
I20260619 09:04:45.203054 17894 raft_consensus.cc:348] T d2d679d7e23540fd88c9df59e74f128d P cb22cee68ee644ac9571f9ccb2c91190 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5920513c169740c1905c47c275c337fa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38613 } } peers { permanent_uuid: "9880f9b2fbcd4a509532112267e1d25e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41383 } } peers { permanent_uuid: "cb22cee68ee644ac9571f9ccb2c91190" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44461 } }
I20260619 09:04:45.203208 17894 raft_consensus.cc:374] T d2d679d7e23540fd88c9df59e74f128d P cb22cee68ee644ac9571f9ccb2c91190 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 09:04:45.203209 17895 raft_consensus.cc:374] T d2d679d7e23540fd88c9df59e74f128d P 5920513c169740c1905c47c275c337fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 09:04:45.203246 17894 raft_consensus.cc:729] T d2d679d7e23540fd88c9df59e74f128d P cb22cee68ee644ac9571f9ccb2c91190 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb22cee68ee644ac9571f9ccb2c91190, State: Initialized, Role: FOLLOWER
I20260619 09:04:45.203250 17895 raft_consensus.cc:729] T d2d679d7e23540fd88c9df59e74f128d P 5920513c169740c1905c47c275c337fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5920513c169740c1905c47c275c337fa, State: Initialized, Role: FOLLOWER
I20260619 09:04:45.203387 17894 consensus_queue.cc:260] T d2d679d7e23540fd88c9df59e74f128d P cb22cee68ee644ac9571f9ccb2c91190 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5920513c169740c1905c47c275c337fa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38613 } } peers { permanent_uuid: "9880f9b2fbcd4a509532112267e1d25e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41383 } } peers { permanent_uuid: "cb22cee68ee644ac9571f9ccb2c91190" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44461 } }
I20260619 09:04:45.203387 17895 consensus_queue.cc:260] T d2d679d7e23540fd88c9df59e74f128d P 5920513c169740c1905c47c275c337fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5920513c169740c1905c47c275c337fa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38613 } } peers { permanent_uuid: "9880f9b2fbcd4a509532112267e1d25e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41383 } } peers { permanent_uuid: "cb22cee68ee644ac9571f9ccb2c91190" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44461 } }
I20260619 09:04:45.203557 17898 tablet_bootstrap.cc:654] T d2d679d7e23540fd88c9df59e74f128d P 9880f9b2fbcd4a509532112267e1d25e: Neither blocks nor log segments found. Creating new log.
I20260619 09:04:45.203661 17895 ts_tablet_manager.cc:1428] T d2d679d7e23540fd88c9df59e74f128d P 5920513c169740c1905c47c275c337fa: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260619 09:04:45.203661 17894 ts_tablet_manager.cc:1428] T d2d679d7e23540fd88c9df59e74f128d P cb22cee68ee644ac9571f9ccb2c91190: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260619 09:04:45.203790 17612 heartbeater.cc:503] Master 127.0.98.62:44537 was elected leader, sending a full tablet report...
I20260619 09:04:45.204046 17880 heartbeater.cc:503] Master 127.0.98.62:44537 was elected leader, sending a full tablet report...
I20260619 09:04:45.204149 17898 log.cc:824] T d2d679d7e23540fd88c9df59e74f128d P 9880f9b2fbcd4a509532112267e1d25e: Log is configured to *not* fsync() on all Append() calls
I20260619 09:04:45.204902 17898 tablet_bootstrap.cc:492] T d2d679d7e23540fd88c9df59e74f128d P 9880f9b2fbcd4a509532112267e1d25e: No bootstrap required, opened a new log
I20260619 09:04:45.204967 17898 ts_tablet_manager.cc:1397] T d2d679d7e23540fd88c9df59e74f128d P 9880f9b2fbcd4a509532112267e1d25e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 09:04:45.206357 17898 raft_consensus.cc:348] T d2d679d7e23540fd88c9df59e74f128d P 9880f9b2fbcd4a509532112267e1d25e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5920513c169740c1905c47c275c337fa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38613 } } peers { permanent_uuid: "9880f9b2fbcd4a509532112267e1d25e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41383 } } peers { permanent_uuid: "cb22cee68ee644ac9571f9ccb2c91190" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44461 } }
I20260619 09:04:45.206489 17898 raft_consensus.cc:374] T d2d679d7e23540fd88c9df59e74f128d P 9880f9b2fbcd4a509532112267e1d25e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 09:04:45.206518 17898 raft_consensus.cc:729] T d2d679d7e23540fd88c9df59e74f128d P 9880f9b2fbcd4a509532112267e1d25e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9880f9b2fbcd4a509532112267e1d25e, State: Initialized, Role: FOLLOWER
I20260619 09:04:45.206629 17898 consensus_queue.cc:260] T d2d679d7e23540fd88c9df59e74f128d P 9880f9b2fbcd4a509532112267e1d25e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5920513c169740c1905c47c275c337fa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38613 } } peers { permanent_uuid: "9880f9b2fbcd4a509532112267e1d25e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41383 } } peers { permanent_uuid: "cb22cee68ee644ac9571f9ccb2c91190" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44461 } }
I20260619 09:04:45.206931 17746 heartbeater.cc:503] Master 127.0.98.62:44537 was elected leader, sending a full tablet report...
I20260619 09:04:45.206938 17898 ts_tablet_manager.cc:1428] T d2d679d7e23540fd88c9df59e74f128d P 9880f9b2fbcd4a509532112267e1d25e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
W20260619 09:04:45.253104 17747 tablet.cc:2369] T d2d679d7e23540fd88c9df59e74f128d P 9880f9b2fbcd4a509532112267e1d25e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 09:04:45.358134 17899 raft_consensus.cc:482] T d2d679d7e23540fd88c9df59e74f128d P 5920513c169740c1905c47c275c337fa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 09:04:45.358278 17899 raft_consensus.cc:504] T d2d679d7e23540fd88c9df59e74f128d P 5920513c169740c1905c47c275c337fa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5920513c169740c1905c47c275c337fa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38613 } } peers { permanent_uuid: "9880f9b2fbcd4a509532112267e1d25e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41383 } } peers { permanent_uuid: "cb22cee68ee644ac9571f9ccb2c91190" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44461 } }
I20260619 09:04:45.358683 17899 leader_election.cc:288] T d2d679d7e23540fd88c9df59e74f128d P 5920513c169740c1905c47c275c337fa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9880f9b2fbcd4a509532112267e1d25e (127.0.98.2:41383), cb22cee68ee644ac9571f9ccb2c91190 (127.0.98.3:44461)
I20260619 09:04:45.361446 17893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:04:45.358992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39739 (local address 127.0.98.2:41383)
0619 09:04:45.359044 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0619 09:04:45.359046 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:04:45.359162 (+ 116us) server_negotiation.cc:407] Connection header received
0619 09:04:45.359212 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:04:45.359213 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:04:45.359226 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:04:45.359270 (+ 44us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 09:04:45.359404 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:04:45.359864 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:04:45.360810 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:04:45.361156 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:04:45.361214 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:04:45.361224 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0619 09:04:45.361346 (+ 122us) server_negotiation.cc:299] Negotiation successful
0619 09:04:45.361378 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260619 09:04:45.361488 17903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:04:45.358947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41383 (local address 127.0.98.1:39739)
0619 09:04:45.359088 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0619 09:04:45.359098 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 09:04:45.359154 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:04:45.359282 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:04:45.359283 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:04:45.359284 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 09:04:45.359352 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:04:45.359354 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:04:45.359935 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:04:45.359936 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:04:45.360731 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:04:45.360733 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:04:45.361227 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:04:45.361227 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:04:45.361290 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:04:45.361293 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 09:04:45.361337 (+ 44us) client_negotiation.cc:240] Negotiation successful
0619 09:04:45.361375 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"mutex_wait_us":16,"thread_start_us":61,"threads_started":1}
I20260619 09:04:45.361487 17892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:04:45.359057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46065 (local address 127.0.98.3:44461)
0619 09:04:45.359109 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0619 09:04:45.359111 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:04:45.359121 (+ 10us) server_negotiation.cc:407] Connection header received
0619 09:04:45.359180 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:04:45.359180 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:04:45.359189 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:04:45.359238 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 09:04:45.359378 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:04:45.359870 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:04:45.360823 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:04:45.361185 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:04:45.361244 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:04:45.361254 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0619 09:04:45.361389 (+ 135us) server_negotiation.cc:299] Negotiation successful
0619 09:04:45.361424 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260619 09:04:45.361497 17614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:04:45.358947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44461 (local address 127.0.98.1:46065)
0619 09:04:45.359029 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0619 09:04:45.359044 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 09:04:45.359109 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:04:45.359241 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:04:45.359242 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:04:45.359244 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 09:04:45.359315 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:04:45.359317 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:04:45.359946 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:04:45.359947 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:04:45.360731 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:04:45.360733 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:04:45.361259 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:04:45.361260 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:04:45.361327 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:04:45.361330 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 09:04:45.361373 (+ 43us) client_negotiation.cc:240] Negotiation successful
0619 09:04:45.361410 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58,"mutex_wait_us":50}
I20260619 09:04:45.361819 17700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2d679d7e23540fd88c9df59e74f128d" candidate_uuid: "5920513c169740c1905c47c275c337fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9880f9b2fbcd4a509532112267e1d25e" is_pre_election: true
I20260619 09:04:45.361884 17834 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2d679d7e23540fd88c9df59e74f128d" candidate_uuid: "5920513c169740c1905c47c275c337fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb22cee68ee644ac9571f9ccb2c91190" is_pre_election: true
I20260619 09:04:45.361955 17700 raft_consensus.cc:2454] T d2d679d7e23540fd88c9df59e74f128d P 9880f9b2fbcd4a509532112267e1d25e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5920513c169740c1905c47c275c337fa in term 0.
I20260619 09:04:45.362030 17834 raft_consensus.cc:2454] T d2d679d7e23540fd88c9df59e74f128d P cb22cee68ee644ac9571f9ccb2c91190 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5920513c169740c1905c47c275c337fa in term 0.
I20260619 09:04:45.362197 17500 leader_election.cc:302] T d2d679d7e23540fd88c9df59e74f128d P 5920513c169740c1905c47c275c337fa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5920513c169740c1905c47c275c337fa, 9880f9b2fbcd4a509532112267e1d25e; no voters:
I20260619 09:04:45.362329 17899 raft_consensus.cc:2781] T d2d679d7e23540fd88c9df59e74f128d P 5920513c169740c1905c47c275c337fa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 09:04:45.362397 17899 raft_consensus.cc:482] T d2d679d7e23540fd88c9df59e74f128d P 5920513c169740c1905c47c275c337fa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 09:04:45.362414 17899 raft_consensus.cc:3037] T d2d679d7e23540fd88c9df59e74f128d P 5920513c169740c1905c47c275c337fa [term 0 FOLLOWER]: Advancing to term 1
I20260619 09:04:45.363116 17899 raft_consensus.cc:504] T d2d679d7e23540fd88c9df59e74f128d P 5920513c169740c1905c47c275c337fa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5920513c169740c1905c47c275c337fa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38613 } } peers { permanent_uuid: "9880f9b2fbcd4a509532112267e1d25e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41383 } } peers { permanent_uuid: "cb22cee68ee644ac9571f9ccb2c91190" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44461 } }
I20260619 09:04:45.363243 17899 leader_election.cc:288] T d2d679d7e23540fd88c9df59e74f128d P 5920513c169740c1905c47c275c337fa [CANDIDATE]: Term 1 election: Requested vote from peers 9880f9b2fbcd4a509532112267e1d25e (127.0.98.2:41383), cb22cee68ee644ac9571f9ccb2c91190 (127.0.98.3:44461)
I20260619 09:04:45.363365 17700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2d679d7e23540fd88c9df59e74f128d" candidate_uuid: "5920513c169740c1905c47c275c337fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9880f9b2fbcd4a509532112267e1d25e"
I20260619 09:04:45.363382 17834 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2d679d7e23540fd88c9df59e74f128d" candidate_uuid: "5920513c169740c1905c47c275c337fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb22cee68ee644ac9571f9ccb2c91190"
I20260619 09:04:45.363441 17834 raft_consensus.cc:3037] T d2d679d7e23540fd88c9df59e74f128d P cb22cee68ee644ac9571f9ccb2c91190 [term 0 FOLLOWER]: Advancing to term 1
I20260619 09:04:45.363437 17700 raft_consensus.cc:3037] T d2d679d7e23540fd88c9df59e74f128d P 9880f9b2fbcd4a509532112267e1d25e [term 0 FOLLOWER]: Advancing to term 1
I20260619 09:04:45.364108 17834 raft_consensus.cc:2454] T d2d679d7e23540fd88c9df59e74f128d P cb22cee68ee644ac9571f9ccb2c91190 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5920513c169740c1905c47c275c337fa in term 1.
I20260619 09:04:45.364121 17700 raft_consensus.cc:2454] T d2d679d7e23540fd88c9df59e74f128d P 9880f9b2fbcd4a509532112267e1d25e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5920513c169740c1905c47c275c337fa in term 1.
I20260619 09:04:45.364259 17500 leader_election.cc:302] T d2d679d7e23540fd88c9df59e74f128d P 5920513c169740c1905c47c275c337fa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5920513c169740c1905c47c275c337fa, 9880f9b2fbcd4a509532112267e1d25e; no voters:
I20260619 09:04:45.364344 17899 raft_consensus.cc:2781] T d2d679d7e23540fd88c9df59e74f128d P 5920513c169740c1905c47c275c337fa [term 1 FOLLOWER]: Leader election won for term 1
I20260619 09:04:45.364509 17899 raft_consensus.cc:686] T d2d679d7e23540fd88c9df59e74f128d P 5920513c169740c1905c47c275c337fa [term 1 LEADER]: Becoming Leader. State: Replica: 5920513c169740c1905c47c275c337fa, State: Running, Role: LEADER
I20260619 09:04:45.364598 17899 consensus_queue.cc:237] T d2d679d7e23540fd88c9df59e74f128d P 5920513c169740c1905c47c275c337fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5920513c169740c1905c47c275c337fa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38613 } } peers { permanent_uuid: "9880f9b2fbcd4a509532112267e1d25e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41383 } } peers { permanent_uuid: "cb22cee68ee644ac9571f9ccb2c91190" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44461 } }
I20260619 09:04:45.365424 17420 catalog_manager.cc:5495] T d2d679d7e23540fd88c9df59e74f128d P 5920513c169740c1905c47c275c337fa reported cstate change: term changed from 0 to 1, leader changed from <none> to 5920513c169740c1905c47c275c337fa (127.0.98.1). New cstate: current_term: 1 leader_uuid: "5920513c169740c1905c47c275c337fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5920513c169740c1905c47c275c337fa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38613 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9880f9b2fbcd4a509532112267e1d25e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41383 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cb22cee68ee644ac9571f9ccb2c91190" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44461 } health_report { overall_health: UNKNOWN } } }
W20260619 09:04:45.366451 17613 tablet.cc:2369] T d2d679d7e23540fd88c9df59e74f128d P 5920513c169740c1905c47c275c337fa: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 09:04:45.374184 17890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:04:45.372044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41186 (local address 127.0.98.1:38613)
0619 09:04:45.372099 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0619 09:04:45.372100 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 09:04:45.372109 (+ 9us) server_negotiation.cc:407] Connection header received
0619 09:04:45.372136 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:04:45.372137 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:04:45.372155 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:04:45.372222 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 09:04:45.372336 (+ 114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:04:45.372796 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:04:45.373569 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:04:45.373756 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:04:45.373817 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:04:45.373929 (+ 112us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 09:04:45.374022 (+ 93us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 09:04:45.374079 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0619 09:04:45.374095 (+ 16us) server_negotiation.cc:299] Negotiation successful
0619 09:04:45.374111 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260619 09:04:45.375849 17700 raft_consensus.cc:1261] T d2d679d7e23540fd88c9df59e74f128d P 9880f9b2fbcd4a509532112267e1d25e [term 1 FOLLOWER]: Refusing update from remote peer 5920513c169740c1905c47c275c337fa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 09:04:45.375847 17834 raft_consensus.cc:1261] T d2d679d7e23540fd88c9df59e74f128d P cb22cee68ee644ac9571f9ccb2c91190 [term 1 FOLLOWER]: Refusing update from remote peer 5920513c169740c1905c47c275c337fa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 09:04:45.376185 17904 consensus_queue.cc:1035] T d2d679d7e23540fd88c9df59e74f128d P 5920513c169740c1905c47c275c337fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb22cee68ee644ac9571f9ccb2c91190" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44461 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 09:04:45.376307 17899 consensus_queue.cc:1035] T d2d679d7e23540fd88c9df59e74f128d P 5920513c169740c1905c47c275c337fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "9880f9b2fbcd4a509532112267e1d25e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 09:04:45.378914 17908 mvcc.cc:205] Tried to move back new op lower bound from 7298498090497253376 to 7298498090454851584. Current Snapshot: MvccSnapshot[applied={T|T < 7298498090497253376}]
I20260619 09:04:45.379246 17909 mvcc.cc:205] Tried to move back new op lower bound from 7298498090497253376 to 7298498090454851584. Current Snapshot: MvccSnapshot[applied={T|T < 7298498090497253376}]
I20260619 09:04:45.380506 17420 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44010:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 09:04:45.380584 17420 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44010:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 09:04:45.381244 17420 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7359ae3596f043f1980850074eb576e8: Sending DeleteTablet for 3 replicas of tablet d2d679d7e23540fd88c9df59e74f128d
I20260619 09:04:45.381601 17680 tablet_service.cc:1514] Processing DeleteTablet for tablet d2d679d7e23540fd88c9df59e74f128d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 09:04:45 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41278
I20260619 09:04:45.381680 17546 tablet_service.cc:1514] Processing DeleteTablet for tablet d2d679d7e23540fd88c9df59e74f128d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 09:04:45 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41182
I20260619 09:04:45.381692 17814 tablet_service.cc:1514] Processing DeleteTablet for tablet d2d679d7e23540fd88c9df59e74f128d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 09:04:45 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59026
I20260619 09:04:45.381891 17918 tablet_replica.cc:330] T d2d679d7e23540fd88c9df59e74f128d P cb22cee68ee644ac9571f9ccb2c91190: stopping tablet replica
I20260619 09:04:45.381937 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ez5NB/build/release/bin/kudu with pid 17483
I20260619 09:04:45.382377 17918 raft_consensus.cc:2229] T d2d679d7e23540fd88c9df59e74f128d P cb22cee68ee644ac9571f9ccb2c91190 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 09:04:45.382490 17918 raft_consensus.cc:2258] T d2d679d7e23540fd88c9df59e74f128d P cb22cee68ee644ac9571f9ccb2c91190 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 09:04:45.382512 17917 tablet_replica.cc:330] T d2d679d7e23540fd88c9df59e74f128d P 9880f9b2fbcd4a509532112267e1d25e: stopping tablet replica
I20260619 09:04:45.382603 17917 raft_consensus.cc:2229] T d2d679d7e23540fd88c9df59e74f128d P 9880f9b2fbcd4a509532112267e1d25e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 09:04:45.382699 17917 raft_consensus.cc:2258] T d2d679d7e23540fd88c9df59e74f128d P 9880f9b2fbcd4a509532112267e1d25e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 09:04:45.383028 17918 ts_tablet_manager.cc:1905] T d2d679d7e23540fd88c9df59e74f128d P cb22cee68ee644ac9571f9ccb2c91190: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 09:04:45.383100 17917 ts_tablet_manager.cc:1905] T d2d679d7e23540fd88c9df59e74f128d P 9880f9b2fbcd4a509532112267e1d25e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 09:04:45.384624 17918 ts_tablet_manager.cc:1918] T d2d679d7e23540fd88c9df59e74f128d P cb22cee68ee644ac9571f9ccb2c91190: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 09:04:45.384687 17918 log.cc:1196] T d2d679d7e23540fd88c9df59e74f128d P cb22cee68ee644ac9571f9ccb2c91190: Deleting WAL directory at /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781859658134510-392-0/minicluster-data/ts-2/wal/wals/d2d679d7e23540fd88c9df59e74f128d
I20260619 09:04:45.384711 17917 ts_tablet_manager.cc:1918] T d2d679d7e23540fd88c9df59e74f128d P 9880f9b2fbcd4a509532112267e1d25e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 09:04:45.384801 17917 log.cc:1196] T d2d679d7e23540fd88c9df59e74f128d P 9880f9b2fbcd4a509532112267e1d25e: Deleting WAL directory at /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781859658134510-392-0/minicluster-data/ts-1/wal/wals/d2d679d7e23540fd88c9df59e74f128d
I20260619 09:04:45.384975 17918 ts_tablet_manager.cc:1939] T d2d679d7e23540fd88c9df59e74f128d P cb22cee68ee644ac9571f9ccb2c91190: Deleting consensus metadata
I20260619 09:04:45.385044 17917 ts_tablet_manager.cc:1939] T d2d679d7e23540fd88c9df59e74f128d P 9880f9b2fbcd4a509532112267e1d25e: Deleting consensus metadata
I20260619 09:04:45.385360 17408 catalog_manager.cc:4841] TS cb22cee68ee644ac9571f9ccb2c91190 (127.0.98.3:44461): tablet d2d679d7e23540fd88c9df59e74f128d (table test-table [id=e60a08fabcc54181a1af0e5ad723550f]) successfully deleted
I20260619 09:04:45.385371 17406 catalog_manager.cc:4841] TS 9880f9b2fbcd4a509532112267e1d25e (127.0.98.2:41383): tablet d2d679d7e23540fd88c9df59e74f128d (table test-table [id=e60a08fabcc54181a1af0e5ad723550f]) successfully deleted
W20260619 09:04:45.387379 17406 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:38613 (error 108)
I20260619 09:04:45.387600 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ez5NB/build/release/bin/kudu with pid 17617
I20260619 09:04:45.388190 17888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:04:45.387730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38613 (local address 127.0.0.1:41192)
0619 09:04:45.388121 (+ 391us) negotiation.cc:107] Waiting for socket to connect
0619 09:04:45.388157 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":372}
W20260619 09:04:45.388470 17406 catalog_manager.cc:4584] TS 5920513c169740c1905c47c275c337fa (127.0.98.1:38613): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d2d679d7e23540fd88c9df59e74f128d: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38613: connect: Connection refused (error 111)
I20260619 09:04:45.391824 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ez5NB/build/release/bin/kudu with pid 17751
I20260619 09:04:45.395864 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ez5NB/build/release/bin/kudu with pid 17389
2026-06-19T09:04:45Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (1880 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-06-19T09:04:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T09:04:45Z Disabled control of system clock
I20260619 09:04:45.421886 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ez5NB/build/release/bin/kudu
/tmp/dist-test-task4ez5NB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781859658134510-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781859658134510-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781859658134510-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781859658134510-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43841
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:38831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43841
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 09:04:45.500679 17924 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 09:04:45.500880 17924 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 09:04:45.500905 17924 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 09:04:45.501999 17924 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 09:04:45.502038 17924 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 09:04:45.502053 17924 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 09:04:45.502068 17924 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 09:04:45.502084 17924 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 09:04:45.503382 17924 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781859658134510-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781859658134510-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43841
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43841
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781859658134510-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17924
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781859658134510-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 09:04:45.503612 17924 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 09:04:45.503861 17924 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 09:04:45.508351 17932 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:04:45.508327 17930 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:04:45.508339 17929 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:04:45.508584 17924 server_base.cc:1034] running on GCE node
I20260619 09:04:45.508836 17924 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 09:04:45.509078 17924 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 09:04:45.510231 17924 hybrid_clock.cc:648] HybridClock initialized: now 1781859885510215 us; error 29 us; skew 500 ppm
I20260619 09:04:45.511368 17924 webserver.cc:460] Webserver started at http://127.0.98.62:45185/ using document root <none> and password file <none>
I20260619 09:04:45.511586 17924 fs_manager.cc:359] Metadata directory not provided
I20260619 09:04:45.511631 17924 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 09:04:45.511749 17924 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 09:04:45.512746 17924 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781859658134510-392-0/minicluster-data/master-0/data/instance:
uuid: "82d1276b55d040508754219b146ba615"
format_stamp: "Formatted at 2026-06-19 09:04:45 on dist-test-slave-mtdr"
server_key: "b2c87c1849ca3ae3a15ddf0032b7b7c5"
server_key_iv: "86b61be3309af1e3aff4081712f7b52d"
server_key_version: "encryptionkey@0"
I20260619 09:04:45.513074 17924 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781859658134510-392-0/minicluster-data/master-0/wal/instance:
uuid: "82d1276b55d040508754219b146ba615"
format_stamp: "Formatted at 2026-06-19 09:04:45 on dist-test-slave-mtdr"
server_key: "b2c87c1849ca3ae3a15ddf0032b7b7c5"
server_key_iv: "86b61be3309af1e3aff4081712f7b52d"
server_key_version: "encryptionkey@0"
I20260619 09:04:45.514199 17924 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260619 09:04:45.514793 17938 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 09:04:45.514937 17924 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260619 09:04:45.515009 17924 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781859658134510-392-0/minicluster-data/master-0/data,/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781859658134510-392-0/minicluster-data/master-0/wal
uuid: "82d1276b55d040508754219b146ba615"
format_stamp: "Formatted at 2026-06-19 09:04:45 on dist-test-slave-mtdr"
server_key: "b2c87c1849ca3ae3a15ddf0032b7b7c5"
server_key_iv: "86b61be3309af1e3aff4081712f7b52d"
server_key_version: "encryptionkey@0"
I20260619 09:04:45.515074 17924 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781859658134510-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781859658134510-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781859658134510-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 09:04:45.535775 17924 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 09:04:45.536109 17924 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 09:04:45.536240 17924 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 09:04:45.539853 17924 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43841
I20260619 09:04:45.539892 17990 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43841 every 8 connection(s)
I20260619 09:04:45.540195 17924 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781859658134510-392-0/minicluster-data/master-0/data/info.pb
I20260619 09:04:45.541077 17991 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 09:04:45.543181 17991 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 82d1276b55d040508754219b146ba615: Bootstrap starting.
I20260619 09:04:45.543665 17991 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 82d1276b55d040508754219b146ba615: Neither blocks nor log segments found. Creating new log.
I20260619 09:04:45.543893 17991 log.cc:824] T 00000000000000000000000000000000 P 82d1276b55d040508754219b146ba615: Log is configured to *not* fsync() on all Append() calls
I20260619 09:04:45.544526 17991 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 82d1276b55d040508754219b146ba615: No bootstrap required, opened a new log
I20260619 09:04:45.546078 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ez5NB/build/release/bin/kudu as pid 17924
I20260619 09:04:45.546010 17991 raft_consensus.cc:348] T 00000000000000000000000000000000 P 82d1276b55d040508754219b146ba615 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82d1276b55d040508754219b146ba615" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43841 } }
I20260619 09:04:45.546139 17991 raft_consensus.cc:374] T 00000000000000000000000000000000 P 82d1276b55d040508754219b146ba615 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 09:04:45.546167 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781859658134510-392-0/minicluster-data/master-0/wal/instance
I20260619 09:04:45.546168 17991 raft_consensus.cc:729] T 00000000000000000000000000000000 P 82d1276b55d040508754219b146ba615 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82d1276b55d040508754219b146ba615, State: Initialized, Role: FOLLOWER
I20260619 09:04:45.546296 17991 consensus_queue.cc:260] T 00000000000000000000000000000000 P 82d1276b55d040508754219b146ba615 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82d1276b55d040508754219b146ba615" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43841 } }
I20260619 09:04:45.546360 392 external_mini_cluster.cc:1428] Setting key 98e2563263e010c98b77f52a189d9def
I20260619 09:04:45.546372 17991 raft_consensus.cc:388] T 00000000000000000000000000000000 P 82d1276b55d040508754219b146ba615 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 09:04:45.546403 17991 raft_consensus.cc:482] T 00000000000000000000000000000000 P 82d1276b55d040508754219b146ba615 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 09:04:45.546454 17991 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 82d1276b55d040508754219b146ba615 [term 0 FOLLOWER]: Advancing to term 1
I20260619 09:04:45.547108 17991 raft_consensus.cc:504] T 00000000000000000000000000000000 P 82d1276b55d040508754219b146ba615 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82d1276b55d040508754219b146ba615" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43841 } }
I20260619 09:04:45.547210 17991 leader_election.cc:302] T 00000000000000000000000000000000 P 82d1276b55d040508754219b146ba615 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 82d1276b55d040508754219b146ba615; no voters:
I20260619 09:04:45.547355 17991 leader_election.cc:288] T 00000000000000000000000000000000 P 82d1276b55d040508754219b146ba615 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 09:04:45.547492 17996 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 82d1276b55d040508754219b146ba615 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 09:04:45.547605 17991 sys_catalog.cc:564] T 00000000000000000000000000000000 P 82d1276b55d040508754219b146ba615 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 09:04:45.547659 17996 raft_consensus.cc:686] T 00000000000000000000000000000000 P 82d1276b55d040508754219b146ba615 [term 1 LEADER]: Becoming Leader. State: Replica: 82d1276b55d040508754219b146ba615, State: Running, Role: LEADER
I20260619 09:04:45.547802 17996 consensus_queue.cc:237] T 00000000000000000000000000000000 P 82d1276b55d040508754219b146ba615 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82d1276b55d040508754219b146ba615" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43841 } }
I20260619 09:04:45.548146 17997 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82d1276b55d040508754219b146ba615 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "82d1276b55d040508754219b146ba615" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82d1276b55d040508754219b146ba615" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43841 } } }
I20260619 09:04:45.548216 17997 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82d1276b55d040508754219b146ba615 [sys.catalog]: This master's current role is: LEADER
I20260619 09:04:45.548357 17998 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82d1276b55d040508754219b146ba615 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 82d1276b55d040508754219b146ba615. Latest consensus state: current_term: 1 leader_uuid: "82d1276b55d040508754219b146ba615" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82d1276b55d040508754219b146ba615" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43841 } } }
I20260619 09:04:45.548404 17998 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82d1276b55d040508754219b146ba615 [sys.catalog]: This master's current role is: LEADER
W20260619 09:04:45.549254 18012 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 82d1276b55d040508754219b146ba615: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 09:04:45.549306 18012 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 09:04:45.549346 18005 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 09:04:45.549489 18005 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 09:04:45.550884 18005 catalog_manager.cc:1347] Generated new cluster ID: 7d425491e1224d6395546158e3cd7234
I20260619 09:04:45.550937 18005 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 09:04:45.556293 17995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:04:45.546852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38056 (local address 127.0.98.62:43841)
0619 09:04:45.547108 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0619 09:04:45.547118 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 09:04:45.547152 (+ 34us) server_negotiation.cc:407] Connection header received
0619 09:04:45.547261 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:04:45.547268 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:04:45.547340 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:04:45.547477 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:04:45.547822 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:04:45.549403 (+ 1581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:04:45.550235 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:04:45.554891 (+ 4656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:04:45.555012 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:04:45.555613 (+ 601us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:04:45.555617 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:04:45.555617 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 09:04:45.555816 (+ 199us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:04:45.555857 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 09:04:45.555871 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:04:45.555932 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0619 09:04:45.556003 (+ 71us) server_negotiation.cc:299] Negotiation successful
0619 09:04:45.556096 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":174,"thread_start_us":121,"threads_started":1}
I20260619 09:04:45.559571 18005 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 09:04:45.560058 18005 catalog_manager.cc:1504] Loading token signing keys...
I20260619 09:04:45.566040 18005 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 82d1276b55d040508754219b146ba615: Generated new TSK 0
I20260619 09:04:45.566215 18005 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 09:04:45.568817 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ez5NB/build/release/bin/kudu
/tmp/dist-test-task4ez5NB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781859658134510-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781859658134510-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781859658134510-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781859658134510-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43841
--builtin_ntp_servers=127.0.98.20:38831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 09:04:45.670291 18015 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 09:04:45.670498 18015 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 09:04:45.670521 18015 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 09:04:45.671695 18015 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 09:04:45.671741 18015 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 09:04:45.671766 18015 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 09:04:45.673090 18015 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781859658134510-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781859658134510-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781859658134510-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43841
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18015
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781859658134510-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 09:04:45.673303 18015 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 09:04:45.673601 18015 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 09:04:45.678427 18023 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:04:45.678432 18020 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:04:45.678432 18021 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:04:45.678593 18015 server_base.cc:1034] running on GCE node
I20260619 09:04:45.678975 18015 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 09:04:45.679193 18015 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 09:04:45.680344 18015 hybrid_clock.cc:648] HybridClock initialized: now 1781859885680332 us; error 27 us; skew 500 ppm
I20260619 09:04:45.681484 18015 webserver.cc:460] Webserver started at http://127.0.98.1:43177/ using document root <none> and password file <none>
I20260619 09:04:45.681703 18015 fs_manager.cc:359] Metadata directory not provided
I20260619 09:04:45.681756 18015 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 09:04:45.681892 18015 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 09:04:45.682977 18015 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781859658134510-392-0/minicluster-data/ts-0/data/instance:
uuid: "70066a56bcd34395ba68a14bf05920cf"
format_stamp: "Formatted at 2026-06-19 09:04:45 on dist-test-slave-mtdr"
server_key: "c668d279af2f3a3770c869862cb900a7"
server_key_iv: "912b25825984061158e0aac9b9d751d3"
server_key_version: "encryptionkey@0"
I20260619 09:04:45.683297 18015 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781859658134510-392-0/minicluster-data/ts-0/wal/instance:
uuid: "70066a56bcd34395ba68a14bf05920cf"
format_stamp: "Formatted at 2026-06-19 09:04:45 on dist-test-slave-mtdr"
server_key: "c668d279af2f3a3770c869862cb900a7"
server_key_iv: "912b25825984061158e0aac9b9d751d3"
server_key_version: "encryptionkey@0"
I20260619 09:04:45.684546 18015 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260619 09:04:45.685236 18029 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 09:04:45.685397 18015 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 09:04:45.685475 18015 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781859658134510-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781859658134510-392-0/minicluster-data/ts-0/wal
uuid: "70066a56bcd34395ba68a14bf05920cf"
format_stamp: "Formatted at 2026-06-19 09:04:45 on dist-test-slave-mtdr"
server_key: "c668d279af2f3a3770c869862cb900a7"
server_key_iv: "912b25825984061158e0aac9b9d751d3"
server_key_version: "encryptionkey@0"
I20260619 09:04:45.685532 18015 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781859658134510-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781859658134510-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781859658134510-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 09:04:45.694594 18015 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 09:04:45.694872 18015 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 09:04:45.694993 18015 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 09:04:45.695186 18015 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 09:04:45.695477 18015 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 09:04:45.695508 18015 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 09:04:45.695533 18015 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 09:04:45.695559 18015 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 09:04:45.701589 18015 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46647
I20260619 09:04:45.701630 18142 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46647 every 8 connection(s)
I20260619 09:04:45.701934 18015 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781859658134510-392-0/minicluster-data/ts-0/data/info.pb
I20260619 09:04:45.704095 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ez5NB/build/release/bin/kudu as pid 18015
I20260619 09:04:45.704201 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781859658134510-392-0/minicluster-data/ts-0/wal/instance
I20260619 09:04:45.704422 392 external_mini_cluster.cc:1428] Setting key ec42f8538505101d5ae243ac06932a8d
I20260619 09:04:45.705312 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ez5NB/build/release/bin/kudu
/tmp/dist-test-task4ez5NB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781859658134510-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781859658134510-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781859658134510-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781859658134510-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43841
--builtin_ntp_servers=127.0.98.20:38831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 09:04:45.707489 17995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:04:45.702595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39433 (local address 127.0.98.62:43841)
0619 09:04:45.702649 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0619 09:04:45.702651 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:04:45.702942 (+ 291us) server_negotiation.cc:407] Connection header received
0619 09:04:45.703427 (+ 485us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:04:45.703428 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:04:45.703439 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:04:45.703489 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:04:45.704220 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:04:45.704780 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:04:45.706114 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:04:45.706301 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:04:45.706356 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:04:45.707227 (+ 871us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:04:45.707228 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:04:45.707229 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 09:04:45.707234 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:04:45.707249 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 09:04:45.707254 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:04:45.707310 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0619 09:04:45.707383 (+ 73us) server_negotiation.cc:299] Negotiation successful
0619 09:04:45.707413 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260619 09:04:45.707624 18145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:04:45.702595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43841 (local address 127.0.98.1:39433)
0619 09:04:45.702754 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0619 09:04:45.702859 (+ 105us) client_negotiation.cc:174] Beginning negotiation
0619 09:04:45.703340 (+ 481us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:04:45.703539 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:04:45.703543 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:04:45.703551 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0619 09:04:45.704099 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:04:45.704106 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:04:45.704876 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:04:45.704877 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:04:45.706023 (+ 1146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:04:45.706026 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:04:45.706376 (+ 350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:04:45.706377 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:04:45.706534 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:04:45.707101 (+ 567us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 09:04:45.707102 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 09:04:45.707130 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 09:04:45.707152 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 09:04:45.707313 (+ 161us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 09:04:45.707315 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 09:04:45.707317 (+ 2us) client_negotiation.cc:769] Sending connection context
0619 09:04:45.707364 (+ 47us) client_negotiation.cc:240] Negotiation successful
0619 09:04:45.707440 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":77,"threads_started":1}
I20260619 09:04:45.708055 18143 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43841
I20260619 09:04:45.708139 18143 heartbeater.cc:464] Registering TS with master...
I20260619 09:04:45.708317 18143 heartbeater.cc:511] Master 127.0.98.62:43841 requested a full tablet report, sending...
I20260619 09:04:45.708798 17955 ts_manager.cc:194] Registered new tserver with Master: 70066a56bcd34395ba68a14bf05920cf (127.0.98.1:46647)
I20260619 09:04:45.709386 17955 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:39433
W20260619 09:04:45.783479 18146 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 09:04:45.783741 18146 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 09:04:45.783773 18146 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 09:04:45.785223 18146 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 09:04:45.785267 18146 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 09:04:45.785292 18146 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 09:04:45.786574 18146 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781859658134510-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781859658134510-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781859658134510-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43841
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18146
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781859658134510-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 09:04:45.786765 18146 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 09:04:45.787109 18146 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 09:04:45.791910 18152 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:04:45.792052 18146 server_base.cc:1034] running on GCE node
W20260619 09:04:45.791916 18151 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:04:45.792130 18154 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:04:45.792310 18146 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 09:04:45.792572 18146 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 09:04:45.793725 18146 hybrid_clock.cc:648] HybridClock initialized: now 1781859885793709 us; error 31 us; skew 500 ppm
I20260619 09:04:45.794966 18146 webserver.cc:460] Webserver started at http://127.0.98.2:40063/ using document root <none> and password file <none>
I20260619 09:04:45.795181 18146 fs_manager.cc:359] Metadata directory not provided
I20260619 09:04:45.795228 18146 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 09:04:45.795373 18146 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 09:04:45.796380 18146 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781859658134510-392-0/minicluster-data/ts-1/data/instance:
uuid: "8b7696cde0004c6fa7a9c0bda80612bc"
format_stamp: "Formatted at 2026-06-19 09:04:45 on dist-test-slave-mtdr"
server_key: "76192bb8b8b9a5effe85cef514fe7b2f"
server_key_iv: "73415fa1206cc4dd4f17eacf53f11202"
server_key_version: "encryptionkey@0"
I20260619 09:04:45.796695 18146 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781859658134510-392-0/minicluster-data/ts-1/wal/instance:
uuid: "8b7696cde0004c6fa7a9c0bda80612bc"
format_stamp: "Formatted at 2026-06-19 09:04:45 on dist-test-slave-mtdr"
server_key: "76192bb8b8b9a5effe85cef514fe7b2f"
server_key_iv: "73415fa1206cc4dd4f17eacf53f11202"
server_key_version: "encryptionkey@0"
I20260619 09:04:45.797833 18146 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 09:04:45.798471 18160 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 09:04:45.798641 18146 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 09:04:45.798719 18146 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781859658134510-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781859658134510-392-0/minicluster-data/ts-1/wal
uuid: "8b7696cde0004c6fa7a9c0bda80612bc"
format_stamp: "Formatted at 2026-06-19 09:04:45 on dist-test-slave-mtdr"
server_key: "76192bb8b8b9a5effe85cef514fe7b2f"
server_key_iv: "73415fa1206cc4dd4f17eacf53f11202"
server_key_version: "encryptionkey@0"
I20260619 09:04:45.798782 18146 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781859658134510-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781859658134510-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781859658134510-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 09:04:45.819221 18146 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 09:04:45.819514 18146 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 09:04:45.819634 18146 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 09:04:45.819842 18146 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 09:04:45.820155 18146 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 09:04:45.820190 18146 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 09:04:45.820219 18146 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 09:04:45.820243 18146 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 09:04:45.825526 18146 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40723
I20260619 09:04:45.825580 18273 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40723 every 8 connection(s)
I20260619 09:04:45.825871 18146 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781859658134510-392-0/minicluster-data/ts-1/data/info.pb
I20260619 09:04:45.830149 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ez5NB/build/release/bin/kudu as pid 18146
I20260619 09:04:45.830248 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781859658134510-392-0/minicluster-data/ts-1/wal/instance
I20260619 09:04:45.830461 392 external_mini_cluster.cc:1428] Setting key 5c33019292938fc5d4afe4df3ed45105
I20260619 09:04:45.831071 17995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:04:45.826599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40079 (local address 127.0.98.62:43841)
0619 09:04:45.826660 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0619 09:04:45.826661 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 09:04:45.826943 (+ 282us) server_negotiation.cc:407] Connection header received
0619 09:04:45.827464 (+ 521us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:04:45.827466 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:04:45.827478 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:04:45.827530 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:04:45.828162 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:04:45.828651 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:04:45.829593 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:04:45.829792 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:04:45.829855 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:04:45.830780 (+ 925us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:04:45.830781 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:04:45.830781 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 09:04:45.830786 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:04:45.830801 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 09:04:45.830806 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:04:45.830887 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0619 09:04:45.830964 (+ 77us) server_negotiation.cc:299] Negotiation successful
0619 09:04:45.830994 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260619 09:04:45.831301 18276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:04:45.826615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43841 (local address 127.0.98.2:40079)
0619 09:04:45.826798 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0619 09:04:45.826864 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0619 09:04:45.827367 (+ 503us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:04:45.827595 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:04:45.827598 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:04:45.827607 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0619 09:04:45.828080 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:04:45.828084 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:04:45.828732 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:04:45.828733 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:04:45.829512 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:04:45.829515 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:04:45.829860 (+ 345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:04:45.829860 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:04:45.829984 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:04:45.830629 (+ 645us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 09:04:45.830630 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 09:04:45.830666 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 09:04:45.830693 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 09:04:45.830888 (+ 195us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 09:04:45.830893 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 09:04:45.830896 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 09:04:45.830966 (+ 70us) client_negotiation.cc:240] Negotiation successful
0619 09:04:45.831065 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":81,"threads_started":1}
I20260619 09:04:45.831458 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ez5NB/build/release/bin/kudu
/tmp/dist-test-task4ez5NB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781859658134510-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781859658134510-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781859658134510-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781859658134510-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43841
--builtin_ntp_servers=127.0.98.20:38831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 09:04:45.831724 18274 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43841
I20260619 09:04:45.831820 18274 heartbeater.cc:464] Registering TS with master...
I20260619 09:04:45.832011 18274 heartbeater.cc:511] Master 127.0.98.62:43841 requested a full tablet report, sending...
I20260619 09:04:45.832355 17955 ts_manager.cc:194] Registered new tserver with Master: 8b7696cde0004c6fa7a9c0bda80612bc (127.0.98.2:40723)
I20260619 09:04:45.832777 17955 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:40079
W20260619 09:04:45.909336 18277 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 09:04:45.909528 18277 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 09:04:45.909547 18277 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 09:04:45.910622 18277 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 09:04:45.910652 18277 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 09:04:45.910674 18277 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 09:04:45.912001 18277 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781859658134510-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781859658134510-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781859658134510-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43841
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18277
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781859658134510-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 09:04:45.912173 18277 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 09:04:45.912402 18277 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 09:04:45.917027 18283 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:04:45.917079 18282 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:04:45.917151 18285 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:04:45.917287 18277 server_base.cc:1034] running on GCE node
I20260619 09:04:45.917470 18277 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 09:04:45.917682 18277 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 09:04:45.918853 18277 hybrid_clock.cc:648] HybridClock initialized: now 1781859885918836 us; error 27 us; skew 500 ppm
I20260619 09:04:45.919997 18277 webserver.cc:460] Webserver started at http://127.0.98.3:46841/ using document root <none> and password file <none>
I20260619 09:04:45.920198 18277 fs_manager.cc:359] Metadata directory not provided
I20260619 09:04:45.920246 18277 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 09:04:45.920360 18277 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 09:04:45.921342 18277 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781859658134510-392-0/minicluster-data/ts-2/data/instance:
uuid: "308e9b50da5a459e9930477af7c4928a"
format_stamp: "Formatted at 2026-06-19 09:04:45 on dist-test-slave-mtdr"
server_key: "9ac245f4fe072acce1712c036e7324c0"
server_key_iv: "a1d994c4a70aad73239bdf20c2a4e194"
server_key_version: "encryptionkey@0"
I20260619 09:04:45.921645 18277 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781859658134510-392-0/minicluster-data/ts-2/wal/instance:
uuid: "308e9b50da5a459e9930477af7c4928a"
format_stamp: "Formatted at 2026-06-19 09:04:45 on dist-test-slave-mtdr"
server_key: "9ac245f4fe072acce1712c036e7324c0"
server_key_iv: "a1d994c4a70aad73239bdf20c2a4e194"
server_key_version: "encryptionkey@0"
I20260619 09:04:45.922855 18277 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 09:04:45.923523 18291 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 09:04:45.923686 18277 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 09:04:45.923766 18277 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781859658134510-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781859658134510-392-0/minicluster-data/ts-2/wal
uuid: "308e9b50da5a459e9930477af7c4928a"
format_stamp: "Formatted at 2026-06-19 09:04:45 on dist-test-slave-mtdr"
server_key: "9ac245f4fe072acce1712c036e7324c0"
server_key_iv: "a1d994c4a70aad73239bdf20c2a4e194"
server_key_version: "encryptionkey@0"
I20260619 09:04:45.923838 18277 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781859658134510-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781859658134510-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781859658134510-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 09:04:45.936074 18277 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 09:04:45.936332 18277 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 09:04:45.936450 18277 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 09:04:45.936641 18277 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 09:04:45.936941 18277 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 09:04:45.936975 18277 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 09:04:45.937000 18277 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 09:04:45.937026 18277 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 09:04:45.942238 18277 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33561
I20260619 09:04:45.942307 18404 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33561 every 8 connection(s)
I20260619 09:04:45.942575 18277 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781859658134510-392-0/minicluster-data/ts-2/data/info.pb
I20260619 09:04:45.946058 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ez5NB/build/release/bin/kudu as pid 18277
I20260619 09:04:45.946142 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781859658134510-392-0/minicluster-data/ts-2/wal/instance
I20260619 09:04:45.946323 392 external_mini_cluster.cc:1428] Setting key b0e86fded42d00e6cb5b062944590eea
I20260619 09:04:45.947877 17995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:04:45.943295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50893 (local address 127.0.98.62:43841)
0619 09:04:45.943354 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0619 09:04:45.943356 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:04:45.943551 (+ 195us) server_negotiation.cc:407] Connection header received
0619 09:04:45.944164 (+ 613us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:04:45.944165 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:04:45.944178 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:04:45.944228 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:04:45.944882 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:04:45.945387 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:04:45.946410 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:04:45.946675 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:04:45.946737 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:04:45.947622 (+ 885us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:04:45.947624 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:04:45.947624 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 09:04:45.947628 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:04:45.947644 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 09:04:45.947649 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:04:45.947723 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0619 09:04:45.947772 (+ 49us) server_negotiation.cc:299] Negotiation successful
0619 09:04:45.947797 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260619 09:04:45.948091 18407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:04:45.943265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43841 (local address 127.0.98.3:50893)
0619 09:04:45.943444 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0619 09:04:45.943475 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0619 09:04:45.944006 (+ 531us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:04:45.944303 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:04:45.944307 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:04:45.944317 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0619 09:04:45.944797 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:04:45.944802 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:04:45.945466 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:04:45.945467 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:04:45.946334 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:04:45.946336 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:04:45.946759 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:04:45.946760 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:04:45.946958 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:04:45.947459 (+ 501us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 09:04:45.947460 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 09:04:45.947490 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 09:04:45.947517 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 09:04:45.947723 (+ 206us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 09:04:45.947725 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 09:04:45.947728 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 09:04:45.947786 (+ 58us) client_negotiation.cc:240] Negotiation successful
0619 09:04:45.947876 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":78,"threads_started":1}
I20260619 09:04:45.948464 18405 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43841
I20260619 09:04:45.948546 18405 heartbeater.cc:464] Registering TS with master...
I20260619 09:04:45.948738 18405 heartbeater.cc:511] Master 127.0.98.62:43841 requested a full tablet report, sending...
I20260619 09:04:45.949113 17955 ts_manager.cc:194] Registered new tserver with Master: 308e9b50da5a459e9930477af7c4928a (127.0.98.3:33561)
I20260619 09:04:45.949478 17955 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:50893
I20260619 09:04:45.958117 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 09:04:45.962047 17995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:04:45.959389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38066 (local address 127.0.98.62:43841)
0619 09:04:45.959454 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0619 09:04:45.959456 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:04:45.959516 (+ 60us) server_negotiation.cc:407] Connection header received
0619 09:04:45.959571 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:04:45.959572 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:04:45.959583 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:04:45.959628 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:04:45.959785 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:04:45.960265 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:04:45.961023 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:04:45.961202 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:04:45.961257 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:04:45.961881 (+ 624us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:04:45.961882 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:04:45.961883 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 09:04:45.961886 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:04:45.961896 (+ 10us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 09:04:45.961899 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:04:45.961934 (+ 35us) server_negotiation.cc:1035] Waiting for connection context
0619 09:04:45.961972 (+ 38us) server_negotiation.cc:299] Negotiation successful
0619 09:04:45.961991 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260619 09:04:45.963477 17955 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38066:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 09:04:45.964924 17955 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 09:04:45.971302 18415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:04:45.967100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40723 (local address 127.0.0.1:46656)
0619 09:04:45.967429 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0619 09:04:45.967439 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 09:04:45.967490 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:04:45.967696 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:04:45.967699 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:04:45.967701 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 09:04:45.967763 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:04:45.967765 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:04:45.968953 (+ 1188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:04:45.968954 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:04:45.970097 (+ 1143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:04:45.970099 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:04:45.971001 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:04:45.971003 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:04:45.971085 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:04:45.971100 (+ 15us) client_negotiation.cc:769] Sending connection context
0619 09:04:45.971171 (+ 71us) client_negotiation.cc:240] Negotiation successful
0619 09:04:45.971222 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":311,"mutex_wait_us":2,"thread_start_us":72,"threads_started":1}
I20260619 09:04:45.971405 18416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:04:45.967260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53442 (local address 127.0.98.1:46647)
0619 09:04:45.967829 (+ 569us) server_negotiation.cc:206] Beginning negotiation
0619 09:04:45.967834 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 09:04:45.967842 (+ 8us) server_negotiation.cc:407] Connection header received
0619 09:04:45.967876 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:04:45.967878 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:04:45.967891 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:04:45.967958 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 09:04:45.968128 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:04:45.969196 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:04:45.970363 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:04:45.971062 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:04:45.971147 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:04:45.971225 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0619 09:04:45.971295 (+ 70us) server_negotiation.cc:299] Negotiation successful
0619 09:04:45.971327 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":526,"thread_start_us":79,"threads_started":1}
I20260619 09:04:45.971678 18417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:04:45.967360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37852 (local address 127.0.98.3:33561)
0619 09:04:45.968187 (+ 827us) server_negotiation.cc:206] Beginning negotiation
0619 09:04:45.968191 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 09:04:45.968199 (+ 8us) server_negotiation.cc:407] Connection header received
0619 09:04:45.968231 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:04:45.968233 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:04:45.968246 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:04:45.968325 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 09:04:45.968488 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:04:45.969506 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:04:45.970529 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:04:45.971113 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:04:45.971178 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:04:45.971259 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0619 09:04:45.971554 (+ 295us) server_negotiation.cc:299] Negotiation successful
0619 09:04:45.971591 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":792,"thread_start_us":72,"threads_started":1}
I20260619 09:04:45.971813 18418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:04:45.967368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46656 (local address 127.0.98.2:40723)
0619 09:04:45.967546 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0619 09:04:45.967550 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 09:04:45.967560 (+ 10us) server_negotiation.cc:407] Connection header received
0619 09:04:45.967600 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:04:45.967602 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:04:45.967614 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:04:45.967678 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 09:04:45.967841 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:04:45.968840 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:04:45.970180 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:04:45.970923 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:04:45.970998 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:04:45.971087 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0619 09:04:45.971692 (+ 605us) server_negotiation.cc:299] Negotiation successful
0619 09:04:45.971733 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":78,"threads_started":1}
I20260619 09:04:45.971396 18414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:04:45.967099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46647 (local address 127.0.0.1:53442)
0619 09:04:45.967259 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0619 09:04:45.967294 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0619 09:04:45.967356 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:04:45.968023 (+ 667us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:04:45.968023 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:04:45.968025 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 09:04:45.968064 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:04:45.968065 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:04:45.969291 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:04:45.969292 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:04:45.970271 (+ 979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:04:45.970274 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:04:45.971155 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:04:45.971156 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:04:45.971216 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:04:45.971220 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 09:04:45.971282 (+ 62us) client_negotiation.cc:240] Negotiation successful
0619 09:04:45.971322 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":50,"threads_started":1}
I20260619 09:04:45.971499 18413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:04:45.967112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33561 (local address 127.0.0.1:37852)
0619 09:04:45.967467 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0619 09:04:45.967474 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0619 09:04:45.967509 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:04:45.968368 (+ 859us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:04:45.968368 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:04:45.968371 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 09:04:45.968422 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:04:45.968424 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:04:45.969581 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:04:45.969582 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:04:45.970460 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:04:45.970462 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:04:45.971293 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:04:45.971294 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:04:45.971347 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:04:45.971351 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 09:04:45.971399 (+ 48us) client_negotiation.cc:240] Negotiation successful
0619 09:04:45.971437 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":102,"threads_started":1}
I20260619 09:04:45.972594 18077 tablet_service.cc:1467] Processing CreateTablet for tablet 91dff60551eb483caf0d5d272c18d8ff (DEFAULT_TABLE table=test-table [id=01314d7e91c24781bbd4c8d98d60c851]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 09:04:45.972630 18208 tablet_service.cc:1467] Processing CreateTablet for tablet 91dff60551eb483caf0d5d272c18d8ff (DEFAULT_TABLE table=test-table [id=01314d7e91c24781bbd4c8d98d60c851]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 09:04:45.972700 18339 tablet_service.cc:1467] Processing CreateTablet for tablet 91dff60551eb483caf0d5d272c18d8ff (DEFAULT_TABLE table=test-table [id=01314d7e91c24781bbd4c8d98d60c851]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 09:04:45.972946 18077 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91dff60551eb483caf0d5d272c18d8ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 09:04:45.972946 18208 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91dff60551eb483caf0d5d272c18d8ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 09:04:45.973001 18339 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91dff60551eb483caf0d5d272c18d8ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 09:04:45.975548 18419 tablet_bootstrap.cc:492] T 91dff60551eb483caf0d5d272c18d8ff P 308e9b50da5a459e9930477af7c4928a: Bootstrap starting.
I20260619 09:04:45.976051 18420 tablet_bootstrap.cc:492] T 91dff60551eb483caf0d5d272c18d8ff P 70066a56bcd34395ba68a14bf05920cf: Bootstrap starting.
I20260619 09:04:45.976121 18419 tablet_bootstrap.cc:654] T 91dff60551eb483caf0d5d272c18d8ff P 308e9b50da5a459e9930477af7c4928a: Neither blocks nor log segments found. Creating new log.
I20260619 09:04:45.976284 18421 tablet_bootstrap.cc:492] T 91dff60551eb483caf0d5d272c18d8ff P 8b7696cde0004c6fa7a9c0bda80612bc: Bootstrap starting.
I20260619 09:04:45.976497 18419 log.cc:824] T 91dff60551eb483caf0d5d272c18d8ff P 308e9b50da5a459e9930477af7c4928a: Log is configured to *not* fsync() on all Append() calls
I20260619 09:04:45.976699 18420 tablet_bootstrap.cc:654] T 91dff60551eb483caf0d5d272c18d8ff P 70066a56bcd34395ba68a14bf05920cf: Neither blocks nor log segments found. Creating new log.
I20260619 09:04:45.976842 18421 tablet_bootstrap.cc:654] T 91dff60551eb483caf0d5d272c18d8ff P 8b7696cde0004c6fa7a9c0bda80612bc: Neither blocks nor log segments found. Creating new log.
I20260619 09:04:45.976991 18420 log.cc:824] T 91dff60551eb483caf0d5d272c18d8ff P 70066a56bcd34395ba68a14bf05920cf: Log is configured to *not* fsync() on all Append() calls
I20260619 09:04:45.977109 18421 log.cc:824] T 91dff60551eb483caf0d5d272c18d8ff P 8b7696cde0004c6fa7a9c0bda80612bc: Log is configured to *not* fsync() on all Append() calls
I20260619 09:04:45.977259 18419 tablet_bootstrap.cc:492] T 91dff60551eb483caf0d5d272c18d8ff P 308e9b50da5a459e9930477af7c4928a: No bootstrap required, opened a new log
I20260619 09:04:45.977371 18419 ts_tablet_manager.cc:1397] T 91dff60551eb483caf0d5d272c18d8ff P 308e9b50da5a459e9930477af7c4928a: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260619 09:04:45.977734 18420 tablet_bootstrap.cc:492] T 91dff60551eb483caf0d5d272c18d8ff P 70066a56bcd34395ba68a14bf05920cf: No bootstrap required, opened a new log
I20260619 09:04:45.977815 18420 ts_tablet_manager.cc:1397] T 91dff60551eb483caf0d5d272c18d8ff P 70066a56bcd34395ba68a14bf05920cf: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 09:04:45.979175 18421 tablet_bootstrap.cc:492] T 91dff60551eb483caf0d5d272c18d8ff P 8b7696cde0004c6fa7a9c0bda80612bc: No bootstrap required, opened a new log
I20260619 09:04:45.979233 18421 ts_tablet_manager.cc:1397] T 91dff60551eb483caf0d5d272c18d8ff P 8b7696cde0004c6fa7a9c0bda80612bc: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260619 09:04:45.979858 18419 raft_consensus.cc:348] T 91dff60551eb483caf0d5d272c18d8ff P 308e9b50da5a459e9930477af7c4928a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b7696cde0004c6fa7a9c0bda80612bc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40723 } } peers { permanent_uuid: "70066a56bcd34395ba68a14bf05920cf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46647 } } peers { permanent_uuid: "308e9b50da5a459e9930477af7c4928a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33561 } }
I20260619 09:04:45.980028 18419 raft_consensus.cc:374] T 91dff60551eb483caf0d5d272c18d8ff P 308e9b50da5a459e9930477af7c4928a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 09:04:45.980062 18419 raft_consensus.cc:729] T 91dff60551eb483caf0d5d272c18d8ff P 308e9b50da5a459e9930477af7c4928a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 308e9b50da5a459e9930477af7c4928a, State: Initialized, Role: FOLLOWER
I20260619 09:04:45.980129 18420 raft_consensus.cc:348] T 91dff60551eb483caf0d5d272c18d8ff P 70066a56bcd34395ba68a14bf05920cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b7696cde0004c6fa7a9c0bda80612bc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40723 } } peers { permanent_uuid: "70066a56bcd34395ba68a14bf05920cf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46647 } } peers { permanent_uuid: "308e9b50da5a459e9930477af7c4928a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33561 } }
I20260619 09:04:45.980211 18419 consensus_queue.cc:260] T 91dff60551eb483caf0d5d272c18d8ff P 308e9b50da5a459e9930477af7c4928a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b7696cde0004c6fa7a9c0bda80612bc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40723 } } peers { permanent_uuid: "70066a56bcd34395ba68a14bf05920cf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46647 } } peers { permanent_uuid: "308e9b50da5a459e9930477af7c4928a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33561 } }
I20260619 09:04:45.980268 18420 raft_consensus.cc:374] T 91dff60551eb483caf0d5d272c18d8ff P 70066a56bcd34395ba68a14bf05920cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 09:04:45.980300 18420 raft_consensus.cc:729] T 91dff60551eb483caf0d5d272c18d8ff P 70066a56bcd34395ba68a14bf05920cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70066a56bcd34395ba68a14bf05920cf, State: Initialized, Role: FOLLOWER
I20260619 09:04:45.980406 18420 consensus_queue.cc:260] T 91dff60551eb483caf0d5d272c18d8ff P 70066a56bcd34395ba68a14bf05920cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b7696cde0004c6fa7a9c0bda80612bc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40723 } } peers { permanent_uuid: "70066a56bcd34395ba68a14bf05920cf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46647 } } peers { permanent_uuid: "308e9b50da5a459e9930477af7c4928a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33561 } }
I20260619 09:04:45.980490 18419 ts_tablet_manager.cc:1428] T 91dff60551eb483caf0d5d272c18d8ff P 308e9b50da5a459e9930477af7c4928a: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260619 09:04:45.980597 18405 heartbeater.cc:503] Master 127.0.98.62:43841 was elected leader, sending a full tablet report...
I20260619 09:04:45.980638 18420 ts_tablet_manager.cc:1428] T 91dff60551eb483caf0d5d272c18d8ff P 70066a56bcd34395ba68a14bf05920cf: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260619 09:04:45.980695 18143 heartbeater.cc:503] Master 127.0.98.62:43841 was elected leader, sending a full tablet report...
I20260619 09:04:45.981331 18421 raft_consensus.cc:348] T 91dff60551eb483caf0d5d272c18d8ff P 8b7696cde0004c6fa7a9c0bda80612bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b7696cde0004c6fa7a9c0bda80612bc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40723 } } peers { permanent_uuid: "70066a56bcd34395ba68a14bf05920cf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46647 } } peers { permanent_uuid: "308e9b50da5a459e9930477af7c4928a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33561 } }
I20260619 09:04:45.981491 18421 raft_consensus.cc:374] T 91dff60551eb483caf0d5d272c18d8ff P 8b7696cde0004c6fa7a9c0bda80612bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 09:04:45.981564 18421 raft_consensus.cc:729] T 91dff60551eb483caf0d5d272c18d8ff P 8b7696cde0004c6fa7a9c0bda80612bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b7696cde0004c6fa7a9c0bda80612bc, State: Initialized, Role: FOLLOWER
I20260619 09:04:45.981702 18421 consensus_queue.cc:260] T 91dff60551eb483caf0d5d272c18d8ff P 8b7696cde0004c6fa7a9c0bda80612bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b7696cde0004c6fa7a9c0bda80612bc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40723 } } peers { permanent_uuid: "70066a56bcd34395ba68a14bf05920cf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46647 } } peers { permanent_uuid: "308e9b50da5a459e9930477af7c4928a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33561 } }
I20260619 09:04:45.981962 18421 ts_tablet_manager.cc:1428] T 91dff60551eb483caf0d5d272c18d8ff P 8b7696cde0004c6fa7a9c0bda80612bc: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260619 09:04:45.982002 18274 heartbeater.cc:503] Master 127.0.98.62:43841 was elected leader, sending a full tablet report...
I20260619 09:04:45.995985 18427 raft_consensus.cc:482] T 91dff60551eb483caf0d5d272c18d8ff P 8b7696cde0004c6fa7a9c0bda80612bc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 09:04:45.996078 18427 raft_consensus.cc:504] T 91dff60551eb483caf0d5d272c18d8ff P 8b7696cde0004c6fa7a9c0bda80612bc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b7696cde0004c6fa7a9c0bda80612bc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40723 } } peers { permanent_uuid: "70066a56bcd34395ba68a14bf05920cf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46647 } } peers { permanent_uuid: "308e9b50da5a459e9930477af7c4928a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33561 } }
I20260619 09:04:45.996361 18427 leader_election.cc:288] T 91dff60551eb483caf0d5d272c18d8ff P 8b7696cde0004c6fa7a9c0bda80612bc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 70066a56bcd34395ba68a14bf05920cf (127.0.98.1:46647), 308e9b50da5a459e9930477af7c4928a (127.0.98.3:33561)
I20260619 09:04:45.999577 18276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:04:45.996511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46647 (local address 127.0.98.2:50991)
0619 09:04:45.996562 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0619 09:04:45.996572 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 09:04:45.996609 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:04:45.996729 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:04:45.996738 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:04:45.996740 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 09:04:45.996811 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:04:45.996813 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:04:45.997719 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:04:45.997720 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:04:45.998709 (+ 989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:04:45.998712 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:04:45.999328 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:04:45.999329 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:04:45.999411 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:04:45.999415 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 09:04:45.999466 (+ 51us) client_negotiation.cc:240] Negotiation successful
0619 09:04:45.999505 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260619 09:04:45.999588 18416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:04:45.996566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50991 (local address 127.0.98.1:46647)
0619 09:04:45.996615 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0619 09:04:45.996617 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:04:45.996627 (+ 10us) server_negotiation.cc:407] Connection header received
0619 09:04:45.996663 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:04:45.996663 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:04:45.996675 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:04:45.996724 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 09:04:45.996950 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:04:45.997613 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:04:45.998837 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:04:45.999251 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:04:45.999319 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:04:45.999330 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0619 09:04:45.999484 (+ 154us) server_negotiation.cc:299] Negotiation successful
0619 09:04:45.999534 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260619 09:04:45.999899 18417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:04:45.996635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55457 (local address 127.0.98.3:33561)
0619 09:04:45.996673 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0619 09:04:45.996675 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:04:45.996751 (+ 76us) server_negotiation.cc:407] Connection header received
0619 09:04:45.996804 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:04:45.996805 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:04:45.996813 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:04:45.996871 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 09:04:45.997101 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:04:45.997691 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:04:45.999014 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:04:45.999539 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:04:45.999585 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:04:45.999594 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0619 09:04:45.999802 (+ 208us) server_negotiation.cc:299] Negotiation successful
0619 09:04:45.999842 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260619 09:04:45.999897 18428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:04:45.996529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33561 (local address 127.0.98.2:55457)
0619 09:04:45.996691 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0619 09:04:45.996698 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0619 09:04:45.996743 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:04:45.996946 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:04:45.996947 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:04:45.996949 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 09:04:45.997020 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:04:45.997022 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:04:45.997782 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:04:45.997783 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:04:45.998917 (+ 1134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:04:45.998920 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:04:45.999654 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:04:45.999654 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:04:45.999729 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:04:45.999733 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 09:04:45.999782 (+ 49us) client_negotiation.cc:240] Negotiation successful
0619 09:04:45.999816 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":95,"threads_started":1}
I20260619 09:04:46.000322 18359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91dff60551eb483caf0d5d272c18d8ff" candidate_uuid: "8b7696cde0004c6fa7a9c0bda80612bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "308e9b50da5a459e9930477af7c4928a" is_pre_election: true
I20260619 09:04:46.000391 18097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91dff60551eb483caf0d5d272c18d8ff" candidate_uuid: "8b7696cde0004c6fa7a9c0bda80612bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "70066a56bcd34395ba68a14bf05920cf" is_pre_election: true
I20260619 09:04:46.000473 18359 raft_consensus.cc:2454] T 91dff60551eb483caf0d5d272c18d8ff P 308e9b50da5a459e9930477af7c4928a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8b7696cde0004c6fa7a9c0bda80612bc in term 0.
I20260619 09:04:46.000543 18097 raft_consensus.cc:2454] T 91dff60551eb483caf0d5d272c18d8ff P 70066a56bcd34395ba68a14bf05920cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8b7696cde0004c6fa7a9c0bda80612bc in term 0.
I20260619 09:04:46.000694 18164 leader_election.cc:302] T 91dff60551eb483caf0d5d272c18d8ff P 8b7696cde0004c6fa7a9c0bda80612bc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 308e9b50da5a459e9930477af7c4928a, 8b7696cde0004c6fa7a9c0bda80612bc; no voters:
I20260619 09:04:46.000825 18427 raft_consensus.cc:2781] T 91dff60551eb483caf0d5d272c18d8ff P 8b7696cde0004c6fa7a9c0bda80612bc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 09:04:46.000890 18427 raft_consensus.cc:482] T 91dff60551eb483caf0d5d272c18d8ff P 8b7696cde0004c6fa7a9c0bda80612bc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 09:04:46.000926 18427 raft_consensus.cc:3037] T 91dff60551eb483caf0d5d272c18d8ff P 8b7696cde0004c6fa7a9c0bda80612bc [term 0 FOLLOWER]: Advancing to term 1
I20260619 09:04:46.001806 18427 raft_consensus.cc:504] T 91dff60551eb483caf0d5d272c18d8ff P 8b7696cde0004c6fa7a9c0bda80612bc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b7696cde0004c6fa7a9c0bda80612bc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40723 } } peers { permanent_uuid: "70066a56bcd34395ba68a14bf05920cf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46647 } } peers { permanent_uuid: "308e9b50da5a459e9930477af7c4928a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33561 } }
I20260619 09:04:46.001931 18427 leader_election.cc:288] T 91dff60551eb483caf0d5d272c18d8ff P 8b7696cde0004c6fa7a9c0bda80612bc [CANDIDATE]: Term 1 election: Requested vote from peers 70066a56bcd34395ba68a14bf05920cf (127.0.98.1:46647), 308e9b50da5a459e9930477af7c4928a (127.0.98.3:33561)
I20260619 09:04:46.002056 18359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91dff60551eb483caf0d5d272c18d8ff" candidate_uuid: "8b7696cde0004c6fa7a9c0bda80612bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "308e9b50da5a459e9930477af7c4928a"
I20260619 09:04:46.002142 18359 raft_consensus.cc:3037] T 91dff60551eb483caf0d5d272c18d8ff P 308e9b50da5a459e9930477af7c4928a [term 0 FOLLOWER]: Advancing to term 1
I20260619 09:04:46.002112 18097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91dff60551eb483caf0d5d272c18d8ff" candidate_uuid: "8b7696cde0004c6fa7a9c0bda80612bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "70066a56bcd34395ba68a14bf05920cf"
I20260619 09:04:46.002354 18097 raft_consensus.cc:3037] T 91dff60551eb483caf0d5d272c18d8ff P 70066a56bcd34395ba68a14bf05920cf [term 0 FOLLOWER]: Advancing to term 1
I20260619 09:04:46.003013 18359 raft_consensus.cc:2454] T 91dff60551eb483caf0d5d272c18d8ff P 308e9b50da5a459e9930477af7c4928a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b7696cde0004c6fa7a9c0bda80612bc in term 1.
I20260619 09:04:46.003155 18164 leader_election.cc:302] T 91dff60551eb483caf0d5d272c18d8ff P 8b7696cde0004c6fa7a9c0bda80612bc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 308e9b50da5a459e9930477af7c4928a, 8b7696cde0004c6fa7a9c0bda80612bc; no voters:
I20260619 09:04:46.003244 18097 raft_consensus.cc:2454] T 91dff60551eb483caf0d5d272c18d8ff P 70066a56bcd34395ba68a14bf05920cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b7696cde0004c6fa7a9c0bda80612bc in term 1.
I20260619 09:04:46.003248 18427 raft_consensus.cc:2781] T 91dff60551eb483caf0d5d272c18d8ff P 8b7696cde0004c6fa7a9c0bda80612bc [term 1 FOLLOWER]: Leader election won for term 1
I20260619 09:04:46.003386 18427 raft_consensus.cc:686] T 91dff60551eb483caf0d5d272c18d8ff P 8b7696cde0004c6fa7a9c0bda80612bc [term 1 LEADER]: Becoming Leader. State: Replica: 8b7696cde0004c6fa7a9c0bda80612bc, State: Running, Role: LEADER
I20260619 09:04:46.003536 18427 consensus_queue.cc:237] T 91dff60551eb483caf0d5d272c18d8ff P 8b7696cde0004c6fa7a9c0bda80612bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b7696cde0004c6fa7a9c0bda80612bc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40723 } } peers { permanent_uuid: "70066a56bcd34395ba68a14bf05920cf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46647 } } peers { permanent_uuid: "308e9b50da5a459e9930477af7c4928a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33561 } }
I20260619 09:04:46.004375 17954 catalog_manager.cc:5495] T 91dff60551eb483caf0d5d272c18d8ff P 8b7696cde0004c6fa7a9c0bda80612bc reported cstate change: term changed from 0 to 1, leader changed from <none> to 8b7696cde0004c6fa7a9c0bda80612bc (127.0.98.2). New cstate: current_term: 1 leader_uuid: "8b7696cde0004c6fa7a9c0bda80612bc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b7696cde0004c6fa7a9c0bda80612bc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40723 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "70066a56bcd34395ba68a14bf05920cf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46647 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "308e9b50da5a459e9930477af7c4928a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33561 } health_report { overall_health: UNKNOWN } } }
I20260619 09:04:46.017194 18418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:04:46.015003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46660 (local address 127.0.98.2:40723)
0619 09:04:46.015058 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0619 09:04:46.015060 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:04:46.015068 (+ 8us) server_negotiation.cc:407] Connection header received
0619 09:04:46.015101 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:04:46.015101 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:04:46.015119 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:04:46.015180 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 09:04:46.015318 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:04:46.015775 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:04:46.016484 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:04:46.016696 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:04:46.016757 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:04:46.016884 (+ 127us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 09:04:46.016983 (+ 99us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 09:04:46.017050 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0619 09:04:46.017083 (+ 33us) server_negotiation.cc:299] Negotiation successful
0619 09:04:46.017101 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260619 09:04:46.018962 18359 raft_consensus.cc:1261] T 91dff60551eb483caf0d5d272c18d8ff P 308e9b50da5a459e9930477af7c4928a [term 1 FOLLOWER]: Refusing update from remote peer 8b7696cde0004c6fa7a9c0bda80612bc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 09:04:46.018980 18097 raft_consensus.cc:1261] T 91dff60551eb483caf0d5d272c18d8ff P 70066a56bcd34395ba68a14bf05920cf [term 1 FOLLOWER]: Refusing update from remote peer 8b7696cde0004c6fa7a9c0bda80612bc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 09:04:46.019451 18429 consensus_queue.cc:1035] T 91dff60551eb483caf0d5d272c18d8ff P 8b7696cde0004c6fa7a9c0bda80612bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "70066a56bcd34395ba68a14bf05920cf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 09:04:46.019614 18427 consensus_queue.cc:1035] T 91dff60551eb483caf0d5d272c18d8ff P 8b7696cde0004c6fa7a9c0bda80612bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "308e9b50da5a459e9930477af7c4928a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 09:04:46.022522 18433 mvcc.cc:205] Tried to move back new op lower bound from 7298498093131390976 to 7298498093071560704. Current Snapshot: MvccSnapshot[applied={T|T < 7298498093131390976}]
I20260619 09:04:46.022715 18434 mvcc.cc:205] Tried to move back new op lower bound from 7298498093131390976 to 7298498093071560704. Current Snapshot: MvccSnapshot[applied={T|T < 7298498093131390976}]
I20260619 09:04:46.024410 17955 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:38066:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 09:04:46.024494 17955 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:38066:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 09:04:46.025192 17955 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 82d1276b55d040508754219b146ba615: Sending DeleteTablet for 3 replicas of tablet 91dff60551eb483caf0d5d272c18d8ff
I20260619 09:04:46.025501 18077 tablet_service.cc:1514] Processing DeleteTablet for tablet 91dff60551eb483caf0d5d272c18d8ff with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 09:04:46 UTC) from {username='slave'} at 127.0.0.1:53442
I20260619 09:04:46.025547 18208 tablet_service.cc:1514] Processing DeleteTablet for tablet 91dff60551eb483caf0d5d272c18d8ff with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 09:04:46 UTC) from {username='slave'} at 127.0.0.1:46656
I20260619 09:04:46.025713 18339 tablet_service.cc:1514] Processing DeleteTablet for tablet 91dff60551eb483caf0d5d272c18d8ff with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 09:04:46 UTC) from {username='slave'} at 127.0.0.1:37852
I20260619 09:04:46.025732 18442 tablet_replica.cc:330] T 91dff60551eb483caf0d5d272c18d8ff P 70066a56bcd34395ba68a14bf05920cf: stopping tablet replica
I20260619 09:04:46.025825 18442 raft_consensus.cc:2229] T 91dff60551eb483caf0d5d272c18d8ff P 70066a56bcd34395ba68a14bf05920cf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 09:04:46.025916 18442 raft_consensus.cc:2258] T 91dff60551eb483caf0d5d272c18d8ff P 70066a56bcd34395ba68a14bf05920cf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 09:04:46.026039 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ez5NB/build/release/bin/kudu with pid 18015
I20260619 09:04:46.027413 18444 tablet_replica.cc:330] T 91dff60551eb483caf0d5d272c18d8ff P 308e9b50da5a459e9930477af7c4928a: stopping tablet replica
I20260619 09:04:46.027554 18444 raft_consensus.cc:2229] T 91dff60551eb483caf0d5d272c18d8ff P 308e9b50da5a459e9930477af7c4928a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 09:04:46.027849 18444 raft_consensus.cc:2258] T 91dff60551eb483caf0d5d272c18d8ff P 308e9b50da5a459e9930477af7c4928a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 09:04:46.027967 18443 tablet_replica.cc:330] T 91dff60551eb483caf0d5d272c18d8ff P 8b7696cde0004c6fa7a9c0bda80612bc: stopping tablet replica
I20260619 09:04:46.028066 18443 raft_consensus.cc:2229] T 91dff60551eb483caf0d5d272c18d8ff P 8b7696cde0004c6fa7a9c0bda80612bc [term 1 LEADER]: Raft consensus shutting down.
I20260619 09:04:46.028169 18443 raft_consensus.cc:2258] T 91dff60551eb483caf0d5d272c18d8ff P 8b7696cde0004c6fa7a9c0bda80612bc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 09:04:46.028488 18444 ts_tablet_manager.cc:1905] T 91dff60551eb483caf0d5d272c18d8ff P 308e9b50da5a459e9930477af7c4928a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 09:04:46.028597 18443 ts_tablet_manager.cc:1905] T 91dff60551eb483caf0d5d272c18d8ff P 8b7696cde0004c6fa7a9c0bda80612bc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 09:04:46.030201 18444 ts_tablet_manager.cc:1918] T 91dff60551eb483caf0d5d272c18d8ff P 308e9b50da5a459e9930477af7c4928a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 09:04:46.030241 18443 ts_tablet_manager.cc:1918] T 91dff60551eb483caf0d5d272c18d8ff P 8b7696cde0004c6fa7a9c0bda80612bc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 09:04:46.030313 18443 log.cc:1196] T 91dff60551eb483caf0d5d272c18d8ff P 8b7696cde0004c6fa7a9c0bda80612bc: Deleting WAL directory at /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781859658134510-392-0/minicluster-data/ts-1/wal/wals/91dff60551eb483caf0d5d272c18d8ff
I20260619 09:04:46.030395 18444 log.cc:1196] T 91dff60551eb483caf0d5d272c18d8ff P 308e9b50da5a459e9930477af7c4928a: Deleting WAL directory at /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781859658134510-392-0/minicluster-data/ts-2/wal/wals/91dff60551eb483caf0d5d272c18d8ff
I20260619 09:04:46.030575 18443 ts_tablet_manager.cc:1939] T 91dff60551eb483caf0d5d272c18d8ff P 8b7696cde0004c6fa7a9c0bda80612bc: Deleting consensus metadata
I20260619 09:04:46.030745 18444 ts_tablet_manager.cc:1939] T 91dff60551eb483caf0d5d272c18d8ff P 308e9b50da5a459e9930477af7c4928a: Deleting consensus metadata
I20260619 09:04:46.030999 17939 catalog_manager.cc:4841] TS 8b7696cde0004c6fa7a9c0bda80612bc (127.0.98.2:40723): tablet 91dff60551eb483caf0d5d272c18d8ff (table test-table [id=01314d7e91c24781bbd4c8d98d60c851]) successfully deleted
I20260619 09:04:46.031225 17942 catalog_manager.cc:4841] TS 308e9b50da5a459e9930477af7c4928a (127.0.98.3:33561): tablet 91dff60551eb483caf0d5d272c18d8ff (table test-table [id=01314d7e91c24781bbd4c8d98d60c851]) successfully deleted
W20260619 09:04:46.031769 17941 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:46647 (error 108)
I20260619 09:04:46.031985 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ez5NB/build/release/bin/kudu with pid 18146
I20260619 09:04:46.032936 18415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:04:46.032119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46647 (local address 127.0.0.1:53448)
0619 09:04:46.032868 (+ 749us) negotiation.cc:107] Waiting for socket to connect
0619 09:04:46.032910 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":726}
W20260619 09:04:46.033393 17941 catalog_manager.cc:4584] TS 70066a56bcd34395ba68a14bf05920cf (127.0.98.1:46647): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 91dff60551eb483caf0d5d272c18d8ff: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46647: connect: Connection refused (error 111)
I20260619 09:04:46.036109 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ez5NB/build/release/bin/kudu with pid 18277
I20260619 09:04:46.040562 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ez5NB/build/release/bin/kudu with pid 17924
2026-06-19T09:04:46Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (643 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781859658134510-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 09:04:46 dist-test-slave-mtdr krb5kdc[18448](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 09:04:46 dist-test-slave-mtdr krb5kdc[18448](info): set up 2 sockets
Jun 19 09:04:46 dist-test-slave-mtdr krb5kdc[18448](info): commencing operation
krb5kdc: starting...
W20260619 09:04:46.990355 392 mini_kdc.cc:121] Time spent starting KDC: real 0.932s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781859658134510-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781859658134510-392-0/krb5kdc/test-user.keytab.
Jun 19 09:04:47 dist-test-slave-mtdr krb5kdc[18448](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859887, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T09:04:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T09:04:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781859658134510-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781859658134510-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 09:04:47.100697 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ez5NB/build/release/bin/kudu
/tmp/dist-test-task4ez5NB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781859658134510-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781859658134510-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781859658134510-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781859658134510-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43633
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781859658134510-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34787
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43633
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781859658134510-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781859658134510-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781859658134510-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 09:04:47.177086 18461 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 09:04:47.177285 18461 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 09:04:47.177315 18461 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 09:04:47.178447 18461 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 09:04:47.178489 18461 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 09:04:47.178506 18461 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 09:04:47.178522 18461 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 09:04:47.178537 18461 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 09:04:47.180013 18461 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34787
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781859658134510-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781859658134510-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43633
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781859658134510-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43633
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781859658134510-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18461
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781859658134510-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 09:04:47.180245 18461 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 09:04:47.180446 18461 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 09:04:47.184962 18467 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:04:47.184973 18466 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:04:47.185101 18461 server_base.cc:1034] running on GCE node
W20260619 09:04:47.184969 18469 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:04:47.185408 18461 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 09:04:47.185662 18461 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 09:04:47.186806 18461 hybrid_clock.cc:648] HybridClock initialized: now 1781859887186797 us; error 35 us; skew 500 ppm
Jun 19 09:04:47 dist-test-slave-mtdr krb5kdc[18448](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859887, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 09:04:47.192158 18461 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 09:04:47.192552 18461 webserver.cc:460] Webserver started at http://127.0.98.62:34463/ using document root <none> and password file <none>
I20260619 09:04:47.192750 18461 fs_manager.cc:359] Metadata directory not provided
I20260619 09:04:47.192798 18461 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 09:04:47.192925 18461 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 09:04:47.193919 18461 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781859658134510-392-0/minicluster-data/master-0/data/instance:
uuid: "23bf14b677cf4be68af716c334dac70e"
format_stamp: "Formatted at 2026-06-19 09:04:47 on dist-test-slave-mtdr"
server_key: "ddd00d1c0838f61ae22a0b9c28967d07"
server_key_iv: "e3cf4210d640d7a40221876c8da8b1de"
server_key_version: "encryptionkey@0"
I20260619 09:04:47.194229 18461 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781859658134510-392-0/minicluster-data/master-0/wal/instance:
uuid: "23bf14b677cf4be68af716c334dac70e"
format_stamp: "Formatted at 2026-06-19 09:04:47 on dist-test-slave-mtdr"
server_key: "ddd00d1c0838f61ae22a0b9c28967d07"
server_key_iv: "e3cf4210d640d7a40221876c8da8b1de"
server_key_version: "encryptionkey@0"
I20260619 09:04:47.195554 18461 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 09:04:47.196346 18476 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 09:04:47.196538 18461 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 09:04:47.196627 18461 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781859658134510-392-0/minicluster-data/master-0/data,/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781859658134510-392-0/minicluster-data/master-0/wal
uuid: "23bf14b677cf4be68af716c334dac70e"
format_stamp: "Formatted at 2026-06-19 09:04:47 on dist-test-slave-mtdr"
server_key: "ddd00d1c0838f61ae22a0b9c28967d07"
server_key_iv: "e3cf4210d640d7a40221876c8da8b1de"
server_key_version: "encryptionkey@0"
I20260619 09:04:47.196689 18461 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781859658134510-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781859658134510-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781859658134510-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 09:04:47.220510 18461 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 09:04:47.222615 18461 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 09:04:47.222759 18461 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 09:04:47.226408 18461 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43633
I20260619 09:04:47.226449 18528 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43633 every 8 connection(s)
I20260619 09:04:47.226770 18461 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781859658134510-392-0/minicluster-data/master-0/data/info.pb
I20260619 09:04:47.227644 18529 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 09:04:47.230010 18529 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 23bf14b677cf4be68af716c334dac70e: Bootstrap starting.
I20260619 09:04:47.230571 18529 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 23bf14b677cf4be68af716c334dac70e: Neither blocks nor log segments found. Creating new log.
I20260619 09:04:47.230880 18529 log.cc:824] T 00000000000000000000000000000000 P 23bf14b677cf4be68af716c334dac70e: Log is configured to *not* fsync() on all Append() calls
I20260619 09:04:47.231568 18529 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 23bf14b677cf4be68af716c334dac70e: No bootstrap required, opened a new log
I20260619 09:04:47.233121 18529 raft_consensus.cc:348] T 00000000000000000000000000000000 P 23bf14b677cf4be68af716c334dac70e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23bf14b677cf4be68af716c334dac70e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43633 } }
I20260619 09:04:47.233268 18529 raft_consensus.cc:374] T 00000000000000000000000000000000 P 23bf14b677cf4be68af716c334dac70e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 09:04:47.233291 18529 raft_consensus.cc:729] T 00000000000000000000000000000000 P 23bf14b677cf4be68af716c334dac70e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23bf14b677cf4be68af716c334dac70e, State: Initialized, Role: FOLLOWER
I20260619 09:04:47.233425 18529 consensus_queue.cc:260] T 00000000000000000000000000000000 P 23bf14b677cf4be68af716c334dac70e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23bf14b677cf4be68af716c334dac70e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43633 } }
I20260619 09:04:47.233494 18529 raft_consensus.cc:388] T 00000000000000000000000000000000 P 23bf14b677cf4be68af716c334dac70e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 09:04:47.233515 18529 raft_consensus.cc:482] T 00000000000000000000000000000000 P 23bf14b677cf4be68af716c334dac70e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 09:04:47.233544 18529 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 23bf14b677cf4be68af716c334dac70e [term 0 FOLLOWER]: Advancing to term 1
I20260619 09:04:47.234018 18529 raft_consensus.cc:504] T 00000000000000000000000000000000 P 23bf14b677cf4be68af716c334dac70e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23bf14b677cf4be68af716c334dac70e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43633 } }
I20260619 09:04:47.234103 18529 leader_election.cc:302] T 00000000000000000000000000000000 P 23bf14b677cf4be68af716c334dac70e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 23bf14b677cf4be68af716c334dac70e; no voters:
I20260619 09:04:47.234239 18529 leader_election.cc:288] T 00000000000000000000000000000000 P 23bf14b677cf4be68af716c334dac70e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 09:04:47.234283 18532 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 23bf14b677cf4be68af716c334dac70e [term 1 FOLLOWER]: Leader election won for term 1
I20260619 09:04:47.234447 18532 raft_consensus.cc:686] T 00000000000000000000000000000000 P 23bf14b677cf4be68af716c334dac70e [term 1 LEADER]: Becoming Leader. State: Replica: 23bf14b677cf4be68af716c334dac70e, State: Running, Role: LEADER
I20260619 09:04:47.234459 18529 sys_catalog.cc:564] T 00000000000000000000000000000000 P 23bf14b677cf4be68af716c334dac70e [sys.catalog]: configured and running, proceeding with master startup.
I20260619 09:04:47.234551 18532 consensus_queue.cc:237] T 00000000000000000000000000000000 P 23bf14b677cf4be68af716c334dac70e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23bf14b677cf4be68af716c334dac70e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43633 } }
I20260619 09:04:47.235018 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ez5NB/build/release/bin/kudu as pid 18461
I20260619 09:04:47.235047 18534 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23bf14b677cf4be68af716c334dac70e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 23bf14b677cf4be68af716c334dac70e. Latest consensus state: current_term: 1 leader_uuid: "23bf14b677cf4be68af716c334dac70e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23bf14b677cf4be68af716c334dac70e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43633 } } }
I20260619 09:04:47.235114 18534 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23bf14b677cf4be68af716c334dac70e [sys.catalog]: This master's current role is: LEADER
I20260619 09:04:47.235123 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781859658134510-392-0/minicluster-data/master-0/wal/instance
I20260619 09:04:47.235328 392 external_mini_cluster.cc:1428] Setting key f7fa27362212dc30c80021b602bc572d
I20260619 09:04:47.235435 18533 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23bf14b677cf4be68af716c334dac70e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "23bf14b677cf4be68af716c334dac70e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23bf14b677cf4be68af716c334dac70e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43633 } } }
I20260619 09:04:47.235504 18533 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23bf14b677cf4be68af716c334dac70e [sys.catalog]: This master's current role is: LEADER
I20260619 09:04:47.235626 18541 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 09:04:47.236166 18541 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 09:04:47.238485 18541 catalog_manager.cc:1347] Generated new cluster ID: 2d9b277e505348579b4cc2497f7a864c
I20260619 09:04:47.238539 18541 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 19 09:04:47 dist-test-slave-mtdr krb5kdc[18448](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859887, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 09:04:47.244736 18546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:04:47.235762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35416 (local address 127.0.98.62:43633)
0619 09:04:47.235961 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0619 09:04:47.235971 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 09:04:47.236008 (+ 37us) server_negotiation.cc:407] Connection header received
0619 09:04:47.236125 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:04:47.236130 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:04:47.236183 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:04:47.236915 (+ 732us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:04:47.237012 (+ 97us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:04:47.238347 (+ 1335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:04:47.239211 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:04:47.239674 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:04:47.239771 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:04:47.241849 (+ 2078us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:04:47.241852 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:04:47.241853 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:04:47.241863 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:04:47.243361 (+ 1498us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:04:47.243658 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:04:47.243661 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:04:47.243661 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:04:47.243774 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:04:47.243943 (+ 169us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:04:47.243944 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:04:47.243944 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:04:47.244158 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:04:47.244316 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0619 09:04:47.244418 (+ 102us) server_negotiation.cc:299] Negotiation successful
0619 09:04:47.244563 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":57,"threads_started":1}
I20260619 09:04:47.254156 18541 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 09:04:47.254891 18541 catalog_manager.cc:1504] Loading token signing keys...
I20260619 09:04:47.268187 18541 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 23bf14b677cf4be68af716c334dac70e: Generated new TSK 0
I20260619 09:04:47.268375 18541 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781859658134510-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781859658134510-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 09:04:47.294097 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ez5NB/build/release/bin/kudu
/tmp/dist-test-task4ez5NB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781859658134510-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781859658134510-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781859658134510-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781859658134510-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43633
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781859658134510-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34787
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781859658134510-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781859658134510-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781859658134510-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 09:04:47.372251 18555 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 09:04:47.372435 18555 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 09:04:47.372463 18555 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 09:04:47.373618 18555 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 09:04:47.373657 18555 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 09:04:47.373692 18555 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 09:04:47.375174 18555 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34787
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781859658134510-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781859658134510-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781859658134510-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781859658134510-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43633
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18555
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781859658134510-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 09:04:47.375391 18555 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 09:04:47.375582 18555 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 09:04:47.380215 18563 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:04:47.380220 18560 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:04:47.380270 18555 server_base.cc:1034] running on GCE node
W20260619 09:04:47.380241 18561 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:04:47.380811 18555 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 09:04:47.381029 18555 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 09:04:47.382187 18555 hybrid_clock.cc:648] HybridClock initialized: now 1781859887382176 us; error 31 us; skew 500 ppm
Jun 19 09:04:47 dist-test-slave-mtdr krb5kdc[18448](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859887, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 09:04:47.384546 18555 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 09:04:47.384928 18555 webserver.cc:460] Webserver started at http://127.0.98.1:38071/ using document root <none> and password file <none>
I20260619 09:04:47.385105 18555 fs_manager.cc:359] Metadata directory not provided
I20260619 09:04:47.385142 18555 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 09:04:47.385234 18555 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 09:04:47.386154 18555 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781859658134510-392-0/minicluster-data/ts-0/data/instance:
uuid: "292e54f4d4fa478ca0b060c275025ece"
format_stamp: "Formatted at 2026-06-19 09:04:47 on dist-test-slave-mtdr"
server_key: "67c4141e5469c1f32e8754f53de62f17"
server_key_iv: "f3b80f003dd4f23b67ea27e16f39ecb0"
server_key_version: "encryptionkey@0"
I20260619 09:04:47.386430 18555 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781859658134510-392-0/minicluster-data/ts-0/wal/instance:
uuid: "292e54f4d4fa478ca0b060c275025ece"
format_stamp: "Formatted at 2026-06-19 09:04:47 on dist-test-slave-mtdr"
server_key: "67c4141e5469c1f32e8754f53de62f17"
server_key_iv: "f3b80f003dd4f23b67ea27e16f39ecb0"
server_key_version: "encryptionkey@0"
I20260619 09:04:47.387607 18555 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 09:04:47.388206 18570 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 09:04:47.388376 18555 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 09:04:47.388453 18555 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781859658134510-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781859658134510-392-0/minicluster-data/ts-0/wal
uuid: "292e54f4d4fa478ca0b060c275025ece"
format_stamp: "Formatted at 2026-06-19 09:04:47 on dist-test-slave-mtdr"
server_key: "67c4141e5469c1f32e8754f53de62f17"
server_key_iv: "f3b80f003dd4f23b67ea27e16f39ecb0"
server_key_version: "encryptionkey@0"
I20260619 09:04:47.388502 18555 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781859658134510-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781859658134510-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781859658134510-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 09:04:47.410770 18555 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 09:04:47.412876 18555 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 09:04:47.412989 18555 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 09:04:47.413175 18555 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 09:04:47.413475 18555 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 09:04:47.413508 18555 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 09:04:47.413530 18555 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 09:04:47.413553 18555 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 09:04:47.419070 18555 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34851
I20260619 09:04:47.419140 18683 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34851 every 8 connection(s)
I20260619 09:04:47.419430 18555 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781859658134510-392-0/minicluster-data/ts-0/data/info.pb
Jun 19 09:04:47 dist-test-slave-mtdr krb5kdc[18448](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859887, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 09:04:47.428108 18546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:04:47.420170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33037 (local address 127.0.98.62:43633)
0619 09:04:47.420213 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0619 09:04:47.420215 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:04:47.420421 (+ 206us) server_negotiation.cc:407] Connection header received
0619 09:04:47.421347 (+ 926us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:04:47.421349 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:04:47.421365 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:04:47.421426 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:04:47.422038 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:04:47.422550 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:04:47.423589 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:04:47.423788 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:04:47.423844 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:04:47.425815 (+ 1971us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:04:47.425817 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:04:47.425817 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:04:47.425825 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:04:47.427034 (+ 1209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:04:47.427393 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:04:47.427394 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:04:47.427394 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:04:47.427440 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:04:47.427624 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:04:47.427624 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:04:47.427625 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:04:47.427721 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:04:47.427767 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0619 09:04:47.427975 (+ 208us) server_negotiation.cc:299] Negotiation successful
0619 09:04:47.428043 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260619 09:04:47.428279 18686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:04:47.420140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43633 (local address 127.0.98.1:33037)
0619 09:04:47.420297 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0619 09:04:47.420338 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 09:04:47.421233 (+ 895us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:04:47.421489 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:04:47.421493 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:04:47.421747 (+ 254us) client_negotiation.cc:189] Negotiated authn=SASL
0619 09:04:47.421947 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:04:47.421951 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:04:47.422631 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:04:47.422631 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:04:47.423505 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:04:47.423507 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:04:47.423860 (+ 353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:04:47.423861 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:04:47.423980 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:04:47.424394 (+ 414us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 09:04:47.424402 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 09:04:47.425730 (+ 1328us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 09:04:47.427101 (+ 1371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:04:47.427107 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:04:47.427109 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:04:47.427332 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:04:47.427486 (+ 154us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:04:47.427486 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:04:47.427487 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:04:47.427579 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:04:47.427765 (+ 186us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 09:04:47.427768 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 09:04:47.427880 (+ 112us) client_negotiation.cc:769] Sending connection context
0619 09:04:47.427957 (+ 77us) client_negotiation.cc:240] Negotiation successful
0619 09:04:47.428074 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":87,"threads_started":1}
I20260619 09:04:47.428599 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ez5NB/build/release/bin/kudu as pid 18555
I20260619 09:04:47.428707 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781859658134510-392-0/minicluster-data/ts-0/wal/instance
I20260619 09:04:47.428815 18684 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43633
I20260619 09:04:47.428917 392 external_mini_cluster.cc:1428] Setting key 4dee3e347e43ebd904ad7edf17cc053d
I20260619 09:04:47.428921 18684 heartbeater.cc:464] Registering TS with master...
I20260619 09:04:47.429090 18684 heartbeater.cc:511] Master 127.0.98.62:43633 requested a full tablet report, sending...
I20260619 09:04:47.429610 18492 ts_manager.cc:194] Registered new tserver with Master: 292e54f4d4fa478ca0b060c275025ece (127.0.98.1:34851)
I20260619 09:04:47.430207 18492 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:33037
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781859658134510-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781859658134510-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 09:04:47.451745 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ez5NB/build/release/bin/kudu
/tmp/dist-test-task4ez5NB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781859658134510-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781859658134510-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781859658134510-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781859658134510-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43633
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781859658134510-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34787
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781859658134510-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781859658134510-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781859658134510-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 09:04:47.529127 18689 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 09:04:47.529325 18689 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 09:04:47.529346 18689 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 09:04:47.530454 18689 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 09:04:47.530488 18689 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 09:04:47.530511 18689 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 09:04:47.531955 18689 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34787
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781859658134510-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781859658134510-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781859658134510-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781859658134510-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43633
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18689
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781859658134510-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 09:04:47.532136 18689 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 09:04:47.532311 18689 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 09:04:47.537235 18697 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:04:47.537326 18689 server_base.cc:1034] running on GCE node
W20260619 09:04:47.537249 18694 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:04:47.537235 18695 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:04:47.537652 18689 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 09:04:47.537873 18689 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 09:04:47.539016 18689 hybrid_clock.cc:648] HybridClock initialized: now 1781859887539002 us; error 29 us; skew 500 ppm
Jun 19 09:04:47 dist-test-slave-mtdr krb5kdc[18448](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859887, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 09:04:47.541399 18689 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 09:04:47.541867 18689 webserver.cc:460] Webserver started at http://127.0.98.2:33577/ using document root <none> and password file <none>
I20260619 09:04:47.542064 18689 fs_manager.cc:359] Metadata directory not provided
I20260619 09:04:47.542114 18689 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 09:04:47.542239 18689 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 09:04:47.543311 18689 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781859658134510-392-0/minicluster-data/ts-1/data/instance:
uuid: "3e080f9e327649bb98ff6d4abf29bf15"
format_stamp: "Formatted at 2026-06-19 09:04:47 on dist-test-slave-mtdr"
server_key: "d459bc59da41c03d219118226c98ab1f"
server_key_iv: "0798c76076c77e9e736ad6b747d5360c"
server_key_version: "encryptionkey@0"
I20260619 09:04:47.543635 18689 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781859658134510-392-0/minicluster-data/ts-1/wal/instance:
uuid: "3e080f9e327649bb98ff6d4abf29bf15"
format_stamp: "Formatted at 2026-06-19 09:04:47 on dist-test-slave-mtdr"
server_key: "d459bc59da41c03d219118226c98ab1f"
server_key_iv: "0798c76076c77e9e736ad6b747d5360c"
server_key_version: "encryptionkey@0"
I20260619 09:04:47.544842 18689 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 09:04:47.545523 18704 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 09:04:47.545701 18689 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 09:04:47.545780 18689 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781859658134510-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781859658134510-392-0/minicluster-data/ts-1/wal
uuid: "3e080f9e327649bb98ff6d4abf29bf15"
format_stamp: "Formatted at 2026-06-19 09:04:47 on dist-test-slave-mtdr"
server_key: "d459bc59da41c03d219118226c98ab1f"
server_key_iv: "0798c76076c77e9e736ad6b747d5360c"
server_key_version: "encryptionkey@0"
I20260619 09:04:47.545843 18689 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781859658134510-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781859658134510-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781859658134510-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 09:04:47.567595 18689 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 09:04:47.569588 18689 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 09:04:47.569722 18689 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 09:04:47.569944 18689 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 09:04:47.570236 18689 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 09:04:47.570269 18689 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 09:04:47.570297 18689 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 09:04:47.570312 18689 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 09:04:47.575670 18689 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45955
I20260619 09:04:47.575733 18817 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45955 every 8 connection(s)
I20260619 09:04:47.576007 18689 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781859658134510-392-0/minicluster-data/ts-1/data/info.pb
Jun 19 09:04:47 dist-test-slave-mtdr krb5kdc[18448](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859887, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 09:04:47.584488 18546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:04:47.576647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43025 (local address 127.0.98.62:43633)
0619 09:04:47.576696 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0619 09:04:47.576698 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:04:47.576884 (+ 186us) server_negotiation.cc:407] Connection header received
0619 09:04:47.577746 (+ 862us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:04:47.577748 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:04:47.577759 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:04:47.577821 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:04:47.578386 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:04:47.578928 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:04:47.579969 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:04:47.580154 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:04:47.580204 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:04:47.582126 (+ 1922us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:04:47.582128 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:04:47.582128 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:04:47.582134 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:04:47.583392 (+ 1258us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:04:47.583741 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:04:47.583742 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:04:47.583742 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:04:47.583794 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:04:47.584027 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:04:47.584027 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:04:47.584027 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:04:47.584127 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:04:47.584171 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0619 09:04:47.584372 (+ 201us) server_negotiation.cc:299] Negotiation successful
0619 09:04:47.584427 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260619 09:04:47.584658 18820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:04:47.576614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43633 (local address 127.0.98.2:43025)
0619 09:04:47.576763 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0619 09:04:47.576803 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 09:04:47.577661 (+ 858us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:04:47.577878 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:04:47.577881 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:04:47.578125 (+ 244us) client_negotiation.cc:189] Negotiated authn=SASL
0619 09:04:47.578310 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:04:47.578315 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:04:47.579029 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:04:47.579030 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:04:47.579892 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:04:47.579895 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:04:47.580212 (+ 317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:04:47.580213 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:04:47.580310 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:04:47.580700 (+ 390us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 09:04:47.580708 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 09:04:47.582038 (+ 1330us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 09:04:47.583462 (+ 1424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:04:47.583467 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:04:47.583469 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:04:47.583683 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:04:47.583866 (+ 183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:04:47.583867 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:04:47.583867 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:04:47.583970 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:04:47.584169 (+ 199us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 09:04:47.584172 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 09:04:47.584280 (+ 108us) client_negotiation.cc:769] Sending connection context
0619 09:04:47.584356 (+ 76us) client_negotiation.cc:240] Negotiation successful
0619 09:04:47.584468 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":62,"threads_started":1}
I20260619 09:04:47.585112 18818 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43633
I20260619 09:04:47.585206 18818 heartbeater.cc:464] Registering TS with master...
I20260619 09:04:47.585366 18818 heartbeater.cc:511] Master 127.0.98.62:43633 requested a full tablet report, sending...
I20260619 09:04:47.585690 18492 ts_manager.cc:194] Registered new tserver with Master: 3e080f9e327649bb98ff6d4abf29bf15 (127.0.98.2:45955)
I20260619 09:04:47.586026 18492 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:43025
I20260619 09:04:47.586215 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ez5NB/build/release/bin/kudu as pid 18689
I20260619 09:04:47.586328 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781859658134510-392-0/minicluster-data/ts-1/wal/instance
I20260619 09:04:47.586506 392 external_mini_cluster.cc:1428] Setting key fe739673f06bea170bbb320846b28135
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781859658134510-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781859658134510-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 09:04:47.609826 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ez5NB/build/release/bin/kudu
/tmp/dist-test-task4ez5NB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781859658134510-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781859658134510-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781859658134510-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781859658134510-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43633
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781859658134510-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34787
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781859658134510-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781859658134510-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781859658134510-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 09:04:47.686619 18823 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 09:04:47.686841 18823 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 09:04:47.686863 18823 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 09:04:47.687980 18823 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 09:04:47.688020 18823 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 09:04:47.688045 18823 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 09:04:47.689448 18823 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34787
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781859658134510-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781859658134510-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781859658134510-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781859658134510-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43633
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18823
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781859658134510-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 09:04:47.689622 18823 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 09:04:47.689802 18823 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 09:04:47.694628 18829 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:04:47.694625 18828 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:04:47.694801 18831 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:04:47.694801 18823 server_base.cc:1034] running on GCE node
I20260619 09:04:47.695096 18823 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 09:04:47.695320 18823 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 09:04:47.696460 18823 hybrid_clock.cc:648] HybridClock initialized: now 1781859887696450 us; error 26 us; skew 500 ppm
Jun 19 09:04:47 dist-test-slave-mtdr krb5kdc[18448](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859887, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 09:04:47.698877 18823 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 09:04:47.699307 18823 webserver.cc:460] Webserver started at http://127.0.98.3:37363/ using document root <none> and password file <none>
I20260619 09:04:47.699496 18823 fs_manager.cc:359] Metadata directory not provided
I20260619 09:04:47.699553 18823 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 09:04:47.699677 18823 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 09:04:47.700740 18823 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781859658134510-392-0/minicluster-data/ts-2/data/instance:
uuid: "2cbc3aa2144f4f3181a0ca9553729cc9"
format_stamp: "Formatted at 2026-06-19 09:04:47 on dist-test-slave-mtdr"
server_key: "3aef4298ba027ba13d6b55acaacbee74"
server_key_iv: "2b508c0391ce47b439e059b3f04ba637"
server_key_version: "encryptionkey@0"
I20260619 09:04:47.701085 18823 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781859658134510-392-0/minicluster-data/ts-2/wal/instance:
uuid: "2cbc3aa2144f4f3181a0ca9553729cc9"
format_stamp: "Formatted at 2026-06-19 09:04:47 on dist-test-slave-mtdr"
server_key: "3aef4298ba027ba13d6b55acaacbee74"
server_key_iv: "2b508c0391ce47b439e059b3f04ba637"
server_key_version: "encryptionkey@0"
I20260619 09:04:47.702320 18823 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 09:04:47.703078 18838 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 09:04:47.703238 18823 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 09:04:47.703325 18823 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781859658134510-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781859658134510-392-0/minicluster-data/ts-2/wal
uuid: "2cbc3aa2144f4f3181a0ca9553729cc9"
format_stamp: "Formatted at 2026-06-19 09:04:47 on dist-test-slave-mtdr"
server_key: "3aef4298ba027ba13d6b55acaacbee74"
server_key_iv: "2b508c0391ce47b439e059b3f04ba637"
server_key_version: "encryptionkey@0"
I20260619 09:04:47.703383 18823 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781859658134510-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781859658134510-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781859658134510-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 09:04:47.711361 18823 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 09:04:47.713027 18823 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 09:04:47.713147 18823 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 09:04:47.713328 18823 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 09:04:47.713591 18823 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 09:04:47.713624 18823 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 09:04:47.713649 18823 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 09:04:47.713672 18823 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 09:04:47.718885 18823 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40369
I20260619 09:04:47.718930 18951 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40369 every 8 connection(s)
I20260619 09:04:47.719233 18823 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781859658134510-392-0/minicluster-data/ts-2/data/info.pb
I20260619 09:04:47.724117 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ez5NB/build/release/bin/kudu as pid 18823
I20260619 09:04:47.724205 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781859658134510-392-0/minicluster-data/ts-2/wal/instance
I20260619 09:04:47.724360 392 external_mini_cluster.cc:1428] Setting key 10c568b29028518b17417f8680e1c45e
Jun 19 09:04:47 dist-test-slave-mtdr krb5kdc[18448](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859887, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 09:04:47.728084 18546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:04:47.719944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42775 (local address 127.0.98.62:43633)
0619 09:04:47.719999 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0619 09:04:47.720001 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:04:47.720179 (+ 178us) server_negotiation.cc:407] Connection header received
0619 09:04:47.721071 (+ 892us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:04:47.721073 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:04:47.721083 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:04:47.721137 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:04:47.721702 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:04:47.722197 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:04:47.723246 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:04:47.723429 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:04:47.723475 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:04:47.725703 (+ 2228us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:04:47.725705 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:04:47.725705 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:04:47.725711 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:04:47.726775 (+ 1064us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:04:47.727188 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:04:47.727189 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:04:47.727189 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:04:47.727240 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:04:47.727487 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:04:47.727488 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:04:47.727489 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:04:47.727629 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:04:47.727693 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0619 09:04:47.727951 (+ 258us) server_negotiation.cc:299] Negotiation successful
0619 09:04:47.728022 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260619 09:04:47.728199 18954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:04:47.719897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43633 (local address 127.0.98.3:42775)
0619 09:04:47.720064 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0619 09:04:47.720103 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 09:04:47.720967 (+ 864us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:04:47.721196 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:04:47.721199 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:04:47.721438 (+ 239us) client_negotiation.cc:189] Negotiated authn=SASL
0619 09:04:47.721621 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:04:47.721626 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:04:47.722274 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:04:47.722275 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:04:47.723171 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:04:47.723173 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:04:47.723518 (+ 345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:04:47.723519 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:04:47.723654 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:04:47.724126 (+ 472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 09:04:47.724133 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0619 09:04:47.725626 (+ 1493us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 09:04:47.726879 (+ 1253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:04:47.726886 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:04:47.726887 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:04:47.727111 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:04:47.727291 (+ 180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:04:47.727292 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:04:47.727292 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:04:47.727397 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:04:47.727693 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 09:04:47.727697 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 09:04:47.727810 (+ 113us) client_negotiation.cc:769] Sending connection context
0619 09:04:47.727898 (+ 88us) client_negotiation.cc:240] Negotiation successful
0619 09:04:47.728022 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":82,"threads_started":1}
I20260619 09:04:47.728683 18952 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43633
I20260619 09:04:47.728775 18952 heartbeater.cc:464] Registering TS with master...
I20260619 09:04:47.728952 18952 heartbeater.cc:511] Master 127.0.98.62:43633 requested a full tablet report, sending...
I20260619 09:04:47.729305 18492 ts_manager.cc:194] Registered new tserver with Master: 2cbc3aa2144f4f3181a0ca9553729cc9 (127.0.98.3:40369)
I20260619 09:04:47.729682 18492 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:42775
I20260619 09:04:47.736018 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 09:04:47.742970 18546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:04:47.737274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35422 (local address 127.0.98.62:43633)
0619 09:04:47.737316 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0619 09:04:47.737318 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:04:47.737404 (+ 86us) server_negotiation.cc:407] Connection header received
0619 09:04:47.737465 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:04:47.737465 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:04:47.737474 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:04:47.737519 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:04:47.738026 (+ 507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:04:47.738491 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:04:47.739233 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:04:47.739429 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:04:47.739481 (+ 52us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:04:47.740629 (+ 1148us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:04:47.740630 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:04:47.740630 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:04:47.740637 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:04:47.741838 (+ 1201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:04:47.742215 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:04:47.742216 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:04:47.742216 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:04:47.742266 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:04:47.742439 (+ 173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:04:47.742440 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:04:47.742440 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:04:47.742571 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:04:47.742660 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0619 09:04:47.742785 (+ 125us) server_negotiation.cc:299] Negotiation successful
0619 09:04:47.742877 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260619 09:04:47.744355 18492 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35422:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 09:04:47.745983 18492 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 09:04:47.752357 18962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:04:47.748143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60076 (local address 127.0.98.1:34851)
0619 09:04:47.748608 (+ 465us) server_negotiation.cc:206] Beginning negotiation
0619 09:04:47.748611 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 09:04:47.748623 (+ 12us) server_negotiation.cc:407] Connection header received
0619 09:04:47.748663 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:04:47.748665 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:04:47.748681 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:04:47.748771 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 09:04:47.749149 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:04:47.749967 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:04:47.751207 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:04:47.751947 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:04:47.752036 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:04:47.752100 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0619 09:04:47.752239 (+ 139us) server_negotiation.cc:299] Negotiation successful
0619 09:04:47.752282 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":429,"thread_start_us":115,"threads_started":1}
I20260619 09:04:47.752349 18961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:04:47.748028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45955 (local address 127.0.0.1:58722)
0619 09:04:47.748313 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0619 09:04:47.748321 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0619 09:04:47.748358 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:04:47.748715 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:04:47.748718 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:04:47.748721 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 09:04:47.748800 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:04:47.748803 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:04:47.750116 (+ 1313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:04:47.750117 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:04:47.751167 (+ 1050us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:04:47.751171 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:04:47.752064 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:04:47.752065 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:04:47.752139 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:04:47.752159 (+ 20us) client_negotiation.cc:769] Sending connection context
0619 09:04:47.752197 (+ 38us) client_negotiation.cc:240] Negotiation successful
0619 09:04:47.752253 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"mutex_wait_us":4,"thread_start_us":50,"threads_started":1}
I20260619 09:04:47.752506 18964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:04:47.748336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47856 (local address 127.0.98.3:40369)
0619 09:04:47.748543 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0619 09:04:47.748548 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 09:04:47.748939 (+ 391us) server_negotiation.cc:407] Connection header received
0619 09:04:47.749201 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:04:47.749204 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:04:47.749221 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:04:47.749303 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 09:04:47.749465 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:04:47.750480 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:04:47.751570 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:04:47.752049 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:04:47.752120 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:04:47.752189 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0619 09:04:47.752387 (+ 198us) server_negotiation.cc:299] Negotiation successful
0619 09:04:47.752429 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":172,"thread_start_us":76,"threads_started":1}
I20260619 09:04:47.752346 18960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:04:47.748028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34851 (local address 127.0.0.1:60076)
0619 09:04:47.748215 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0619 09:04:47.748254 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 09:04:47.748313 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:04:47.749029 (+ 716us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:04:47.749030 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:04:47.749032 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 09:04:47.749068 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:04:47.749070 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:04:47.750056 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:04:47.750057 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:04:47.751112 (+ 1055us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:04:47.751114 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:04:47.752047 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:04:47.752048 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:04:47.752131 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:04:47.752147 (+ 16us) client_negotiation.cc:769] Sending connection context
0619 09:04:47.752214 (+ 67us) client_negotiation.cc:240] Negotiation successful
0619 09:04:47.752259 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":53,"threads_started":1}
I20260619 09:04:47.752619 18963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:04:47.748225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40369 (local address 127.0.0.1:47856)
0619 09:04:47.748869 (+ 644us) negotiation.cc:107] Waiting for socket to connect
0619 09:04:47.748879 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 09:04:47.748937 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:04:47.749310 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:04:47.749311 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:04:47.749313 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 09:04:47.749354 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:04:47.749356 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:04:47.750588 (+ 1232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:04:47.750589 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:04:47.751501 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:04:47.751503 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:04:47.752221 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:04:47.752222 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:04:47.752273 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:04:47.752276 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 09:04:47.752320 (+ 44us) client_negotiation.cc:240] Negotiation successful
0619 09:04:47.752351 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":630,"thread_start_us":47,"threads_started":1}
I20260619 09:04:47.753194 18965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:04:47.748390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58722 (local address 127.0.98.2:45955)
0619 09:04:47.748545 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0619 09:04:47.748549 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 09:04:47.748557 (+ 8us) server_negotiation.cc:407] Connection header received
0619 09:04:47.748588 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:04:47.748590 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:04:47.748601 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:04:47.748705 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 09:04:47.748897 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:04:47.749985 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:04:47.751271 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:04:47.751959 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:04:47.752395 (+ 436us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:04:47.753039 (+ 644us) server_negotiation.cc:1035] Waiting for connection context
0619 09:04:47.753066 (+ 27us) server_negotiation.cc:299] Negotiation successful
0619 09:04:47.753108 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":87,"threads_started":1}
I20260619 09:04:47.753249 18886 tablet_service.cc:1467] Processing CreateTablet for tablet 192827240aa34527bb4e1c523f8791a1 (DEFAULT_TABLE table=test-table [id=66f74f4fcb614a14bac62ab9b667f3a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 09:04:47.753248 18618 tablet_service.cc:1467] Processing CreateTablet for tablet 192827240aa34527bb4e1c523f8791a1 (DEFAULT_TABLE table=test-table [id=66f74f4fcb614a14bac62ab9b667f3a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 09:04:47.753593 18618 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 192827240aa34527bb4e1c523f8791a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 09:04:47.753593 18886 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 192827240aa34527bb4e1c523f8791a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 09:04:47.753921 18752 tablet_service.cc:1467] Processing CreateTablet for tablet 192827240aa34527bb4e1c523f8791a1 (DEFAULT_TABLE table=test-table [id=66f74f4fcb614a14bac62ab9b667f3a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 09:04:47.754209 18752 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 192827240aa34527bb4e1c523f8791a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 09:04:47.756790 18966 tablet_bootstrap.cc:492] T 192827240aa34527bb4e1c523f8791a1 P 2cbc3aa2144f4f3181a0ca9553729cc9: Bootstrap starting.
I20260619 09:04:47.757006 18967 tablet_bootstrap.cc:492] T 192827240aa34527bb4e1c523f8791a1 P 292e54f4d4fa478ca0b060c275025ece: Bootstrap starting.
I20260619 09:04:47.757097 18968 tablet_bootstrap.cc:492] T 192827240aa34527bb4e1c523f8791a1 P 3e080f9e327649bb98ff6d4abf29bf15: Bootstrap starting.
I20260619 09:04:47.757539 18967 tablet_bootstrap.cc:654] T 192827240aa34527bb4e1c523f8791a1 P 292e54f4d4fa478ca0b060c275025ece: Neither blocks nor log segments found. Creating new log.
I20260619 09:04:47.757561 18966 tablet_bootstrap.cc:654] T 192827240aa34527bb4e1c523f8791a1 P 2cbc3aa2144f4f3181a0ca9553729cc9: Neither blocks nor log segments found. Creating new log.
I20260619 09:04:47.757711 18968 tablet_bootstrap.cc:654] T 192827240aa34527bb4e1c523f8791a1 P 3e080f9e327649bb98ff6d4abf29bf15: Neither blocks nor log segments found. Creating new log.
I20260619 09:04:47.758035 18967 log.cc:824] T 192827240aa34527bb4e1c523f8791a1 P 292e54f4d4fa478ca0b060c275025ece: Log is configured to *not* fsync() on all Append() calls
I20260619 09:04:47.758170 18966 log.cc:824] T 192827240aa34527bb4e1c523f8791a1 P 2cbc3aa2144f4f3181a0ca9553729cc9: Log is configured to *not* fsync() on all Append() calls
I20260619 09:04:47.758270 18968 log.cc:824] T 192827240aa34527bb4e1c523f8791a1 P 3e080f9e327649bb98ff6d4abf29bf15: Log is configured to *not* fsync() on all Append() calls
I20260619 09:04:47.758749 18967 tablet_bootstrap.cc:492] T 192827240aa34527bb4e1c523f8791a1 P 292e54f4d4fa478ca0b060c275025ece: No bootstrap required, opened a new log
I20260619 09:04:47.758843 18967 ts_tablet_manager.cc:1397] T 192827240aa34527bb4e1c523f8791a1 P 292e54f4d4fa478ca0b060c275025ece: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260619 09:04:47.759034 18968 tablet_bootstrap.cc:492] T 192827240aa34527bb4e1c523f8791a1 P 3e080f9e327649bb98ff6d4abf29bf15: No bootstrap required, opened a new log
I20260619 09:04:47.759101 18968 ts_tablet_manager.cc:1397] T 192827240aa34527bb4e1c523f8791a1 P 3e080f9e327649bb98ff6d4abf29bf15: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 09:04:47.760744 18967 raft_consensus.cc:348] T 192827240aa34527bb4e1c523f8791a1 P 292e54f4d4fa478ca0b060c275025ece [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e080f9e327649bb98ff6d4abf29bf15" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45955 } } peers { permanent_uuid: "292e54f4d4fa478ca0b060c275025ece" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34851 } } peers { permanent_uuid: "2cbc3aa2144f4f3181a0ca9553729cc9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40369 } }
I20260619 09:04:47.760941 18967 raft_consensus.cc:374] T 192827240aa34527bb4e1c523f8791a1 P 292e54f4d4fa478ca0b060c275025ece [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 09:04:47.760984 18967 raft_consensus.cc:729] T 192827240aa34527bb4e1c523f8791a1 P 292e54f4d4fa478ca0b060c275025ece [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 292e54f4d4fa478ca0b060c275025ece, State: Initialized, Role: FOLLOWER
I20260619 09:04:47.760927 18968 raft_consensus.cc:348] T 192827240aa34527bb4e1c523f8791a1 P 3e080f9e327649bb98ff6d4abf29bf15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e080f9e327649bb98ff6d4abf29bf15" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45955 } } peers { permanent_uuid: "292e54f4d4fa478ca0b060c275025ece" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34851 } } peers { permanent_uuid: "2cbc3aa2144f4f3181a0ca9553729cc9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40369 } }
I20260619 09:04:47.761080 18968 raft_consensus.cc:374] T 192827240aa34527bb4e1c523f8791a1 P 3e080f9e327649bb98ff6d4abf29bf15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 09:04:47.761118 18968 raft_consensus.cc:729] T 192827240aa34527bb4e1c523f8791a1 P 3e080f9e327649bb98ff6d4abf29bf15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e080f9e327649bb98ff6d4abf29bf15, State: Initialized, Role: FOLLOWER
I20260619 09:04:47.761103 18967 consensus_queue.cc:260] T 192827240aa34527bb4e1c523f8791a1 P 292e54f4d4fa478ca0b060c275025ece [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e080f9e327649bb98ff6d4abf29bf15" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45955 } } peers { permanent_uuid: "292e54f4d4fa478ca0b060c275025ece" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34851 } } peers { permanent_uuid: "2cbc3aa2144f4f3181a0ca9553729cc9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40369 } }
I20260619 09:04:47.761216 18968 consensus_queue.cc:260] T 192827240aa34527bb4e1c523f8791a1 P 3e080f9e327649bb98ff6d4abf29bf15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e080f9e327649bb98ff6d4abf29bf15" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45955 } } peers { permanent_uuid: "292e54f4d4fa478ca0b060c275025ece" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34851 } } peers { permanent_uuid: "2cbc3aa2144f4f3181a0ca9553729cc9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40369 } }
I20260619 09:04:47.761392 18967 ts_tablet_manager.cc:1428] T 192827240aa34527bb4e1c523f8791a1 P 292e54f4d4fa478ca0b060c275025ece: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260619 09:04:47.761402 18684 heartbeater.cc:503] Master 127.0.98.62:43633 was elected leader, sending a full tablet report...
I20260619 09:04:47.761452 18968 ts_tablet_manager.cc:1428] T 192827240aa34527bb4e1c523f8791a1 P 3e080f9e327649bb98ff6d4abf29bf15: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260619 09:04:47.761548 18818 heartbeater.cc:503] Master 127.0.98.62:43633 was elected leader, sending a full tablet report...
I20260619 09:04:47.761883 18966 tablet_bootstrap.cc:492] T 192827240aa34527bb4e1c523f8791a1 P 2cbc3aa2144f4f3181a0ca9553729cc9: No bootstrap required, opened a new log
I20260619 09:04:47.761956 18966 ts_tablet_manager.cc:1397] T 192827240aa34527bb4e1c523f8791a1 P 2cbc3aa2144f4f3181a0ca9553729cc9: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260619 09:04:47.763495 18966 raft_consensus.cc:348] T 192827240aa34527bb4e1c523f8791a1 P 2cbc3aa2144f4f3181a0ca9553729cc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e080f9e327649bb98ff6d4abf29bf15" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45955 } } peers { permanent_uuid: "292e54f4d4fa478ca0b060c275025ece" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34851 } } peers { permanent_uuid: "2cbc3aa2144f4f3181a0ca9553729cc9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40369 } }
I20260619 09:04:47.763624 18966 raft_consensus.cc:374] T 192827240aa34527bb4e1c523f8791a1 P 2cbc3aa2144f4f3181a0ca9553729cc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 09:04:47.763653 18966 raft_consensus.cc:729] T 192827240aa34527bb4e1c523f8791a1 P 2cbc3aa2144f4f3181a0ca9553729cc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2cbc3aa2144f4f3181a0ca9553729cc9, State: Initialized, Role: FOLLOWER
I20260619 09:04:47.763747 18966 consensus_queue.cc:260] T 192827240aa34527bb4e1c523f8791a1 P 2cbc3aa2144f4f3181a0ca9553729cc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e080f9e327649bb98ff6d4abf29bf15" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45955 } } peers { permanent_uuid: "292e54f4d4fa478ca0b060c275025ece" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34851 } } peers { permanent_uuid: "2cbc3aa2144f4f3181a0ca9553729cc9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40369 } }
I20260619 09:04:47.764020 18966 ts_tablet_manager.cc:1428] T 192827240aa34527bb4e1c523f8791a1 P 2cbc3aa2144f4f3181a0ca9553729cc9: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260619 09:04:47.764047 18952 heartbeater.cc:503] Master 127.0.98.62:43633 was elected leader, sending a full tablet report...
W20260619 09:04:47.826553 18819 tablet.cc:2369] T 192827240aa34527bb4e1c523f8791a1 P 3e080f9e327649bb98ff6d4abf29bf15: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 09:04:47.920166 18685 tablet.cc:2369] T 192827240aa34527bb4e1c523f8791a1 P 292e54f4d4fa478ca0b060c275025ece: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 09:04:47.969798 18953 tablet.cc:2369] T 192827240aa34527bb4e1c523f8791a1 P 2cbc3aa2144f4f3181a0ca9553729cc9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 09:04:48.022367 18973 raft_consensus.cc:482] T 192827240aa34527bb4e1c523f8791a1 P 3e080f9e327649bb98ff6d4abf29bf15 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 09:04:48.022545 18973 raft_consensus.cc:504] T 192827240aa34527bb4e1c523f8791a1 P 3e080f9e327649bb98ff6d4abf29bf15 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e080f9e327649bb98ff6d4abf29bf15" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45955 } } peers { permanent_uuid: "292e54f4d4fa478ca0b060c275025ece" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34851 } } peers { permanent_uuid: "2cbc3aa2144f4f3181a0ca9553729cc9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40369 } }
I20260619 09:04:48.022953 18973 leader_election.cc:288] T 192827240aa34527bb4e1c523f8791a1 P 3e080f9e327649bb98ff6d4abf29bf15 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 292e54f4d4fa478ca0b060c275025ece (127.0.98.1:34851), 2cbc3aa2144f4f3181a0ca9553729cc9 (127.0.98.3:40369)
I20260619 09:04:48.025664 18962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:04:48.023247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56319 (local address 127.0.98.1:34851)
0619 09:04:48.023295 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0619 09:04:48.023297 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:04:48.023308 (+ 11us) server_negotiation.cc:407] Connection header received
0619 09:04:48.023352 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:04:48.023353 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:04:48.023365 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:04:48.023411 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 09:04:48.023601 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:04:48.024080 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:04:48.025014 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:04:48.025359 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:04:48.025417 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:04:48.025427 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0619 09:04:48.025567 (+ 140us) server_negotiation.cc:299] Negotiation successful
0619 09:04:48.025602 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260619 09:04:48.025664 18820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:04:48.023143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34851 (local address 127.0.98.2:56319)
0619 09:04:48.023216 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0619 09:04:48.023228 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 09:04:48.023293 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:04:48.023461 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:04:48.023462 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:04:48.023464 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 09:04:48.023523 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:04:48.023531 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:04:48.024154 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:04:48.024155 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:04:48.024924 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:04:48.024926 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:04:48.025436 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:04:48.025437 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:04:48.025506 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:04:48.025509 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 09:04:48.025553 (+ 44us) client_negotiation.cc:240] Negotiation successful
0619 09:04:48.025589 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260619 09:04:48.025746 18964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:04:48.023229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34625 (local address 127.0.98.3:40369)
0619 09:04:48.023315 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0619 09:04:48.023317 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:04:48.023462 (+ 145us) server_negotiation.cc:407] Connection header received
0619 09:04:48.023542 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:04:48.023544 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:04:48.023553 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:04:48.023597 (+ 44us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 09:04:48.023733 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:04:48.024185 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:04:48.025068 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:04:48.025428 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:04:48.025484 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:04:48.025494 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0619 09:04:48.025643 (+ 149us) server_negotiation.cc:299] Negotiation successful
0619 09:04:48.025676 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 09:04:48.025766 18975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:04:48.023143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40369 (local address 127.0.98.2:34625)
0619 09:04:48.023378 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0619 09:04:48.023386 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0619 09:04:48.023477 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:04:48.023610 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:04:48.023611 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:04:48.023612 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 09:04:48.023681 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:04:48.023682 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:04:48.024255 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:04:48.024256 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:04:48.024995 (+ 739us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:04:48.024997 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:04:48.025499 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:04:48.025500 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:04:48.025587 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:04:48.025590 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 09:04:48.025633 (+ 43us) client_negotiation.cc:240] Negotiation successful
0619 09:04:48.025670 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":220,"mutex_wait_us":76,"thread_start_us":98,"threads_started":1}
I20260619 09:04:48.025982 18638 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "192827240aa34527bb4e1c523f8791a1" candidate_uuid: "3e080f9e327649bb98ff6d4abf29bf15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "292e54f4d4fa478ca0b060c275025ece" is_pre_election: true
I20260619 09:04:48.026069 18906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "192827240aa34527bb4e1c523f8791a1" candidate_uuid: "3e080f9e327649bb98ff6d4abf29bf15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2cbc3aa2144f4f3181a0ca9553729cc9" is_pre_election: true
I20260619 09:04:48.026122 18638 raft_consensus.cc:2454] T 192827240aa34527bb4e1c523f8791a1 P 292e54f4d4fa478ca0b060c275025ece [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3e080f9e327649bb98ff6d4abf29bf15 in term 0.
I20260619 09:04:48.026206 18906 raft_consensus.cc:2454] T 192827240aa34527bb4e1c523f8791a1 P 2cbc3aa2144f4f3181a0ca9553729cc9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3e080f9e327649bb98ff6d4abf29bf15 in term 0.
I20260619 09:04:48.026345 18705 leader_election.cc:302] T 192827240aa34527bb4e1c523f8791a1 P 3e080f9e327649bb98ff6d4abf29bf15 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 292e54f4d4fa478ca0b060c275025ece, 3e080f9e327649bb98ff6d4abf29bf15; no voters:
I20260619 09:04:48.026475 18973 raft_consensus.cc:2781] T 192827240aa34527bb4e1c523f8791a1 P 3e080f9e327649bb98ff6d4abf29bf15 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 09:04:48.026527 18973 raft_consensus.cc:482] T 192827240aa34527bb4e1c523f8791a1 P 3e080f9e327649bb98ff6d4abf29bf15 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 09:04:48.026544 18973 raft_consensus.cc:3037] T 192827240aa34527bb4e1c523f8791a1 P 3e080f9e327649bb98ff6d4abf29bf15 [term 0 FOLLOWER]: Advancing to term 1
I20260619 09:04:48.027235 18973 raft_consensus.cc:504] T 192827240aa34527bb4e1c523f8791a1 P 3e080f9e327649bb98ff6d4abf29bf15 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e080f9e327649bb98ff6d4abf29bf15" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45955 } } peers { permanent_uuid: "292e54f4d4fa478ca0b060c275025ece" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34851 } } peers { permanent_uuid: "2cbc3aa2144f4f3181a0ca9553729cc9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40369 } }
I20260619 09:04:48.027344 18973 leader_election.cc:288] T 192827240aa34527bb4e1c523f8791a1 P 3e080f9e327649bb98ff6d4abf29bf15 [CANDIDATE]: Term 1 election: Requested vote from peers 292e54f4d4fa478ca0b060c275025ece (127.0.98.1:34851), 2cbc3aa2144f4f3181a0ca9553729cc9 (127.0.98.3:40369)
I20260619 09:04:48.027503 18906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "192827240aa34527bb4e1c523f8791a1" candidate_uuid: "3e080f9e327649bb98ff6d4abf29bf15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2cbc3aa2144f4f3181a0ca9553729cc9"
I20260619 09:04:48.027572 18906 raft_consensus.cc:3037] T 192827240aa34527bb4e1c523f8791a1 P 2cbc3aa2144f4f3181a0ca9553729cc9 [term 0 FOLLOWER]: Advancing to term 1
I20260619 09:04:48.027599 18638 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "192827240aa34527bb4e1c523f8791a1" candidate_uuid: "3e080f9e327649bb98ff6d4abf29bf15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "292e54f4d4fa478ca0b060c275025ece"
I20260619 09:04:48.027679 18638 raft_consensus.cc:3037] T 192827240aa34527bb4e1c523f8791a1 P 292e54f4d4fa478ca0b060c275025ece [term 0 FOLLOWER]: Advancing to term 1
I20260619 09:04:48.028270 18906 raft_consensus.cc:2454] T 192827240aa34527bb4e1c523f8791a1 P 2cbc3aa2144f4f3181a0ca9553729cc9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e080f9e327649bb98ff6d4abf29bf15 in term 1.
I20260619 09:04:48.028334 18638 raft_consensus.cc:2454] T 192827240aa34527bb4e1c523f8791a1 P 292e54f4d4fa478ca0b060c275025ece [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e080f9e327649bb98ff6d4abf29bf15 in term 1.
I20260619 09:04:48.028466 18706 leader_election.cc:302] T 192827240aa34527bb4e1c523f8791a1 P 3e080f9e327649bb98ff6d4abf29bf15 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2cbc3aa2144f4f3181a0ca9553729cc9, 3e080f9e327649bb98ff6d4abf29bf15; no voters:
I20260619 09:04:48.028553 18973 raft_consensus.cc:2781] T 192827240aa34527bb4e1c523f8791a1 P 3e080f9e327649bb98ff6d4abf29bf15 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 09:04:48.028687 18973 raft_consensus.cc:686] T 192827240aa34527bb4e1c523f8791a1 P 3e080f9e327649bb98ff6d4abf29bf15 [term 1 LEADER]: Becoming Leader. State: Replica: 3e080f9e327649bb98ff6d4abf29bf15, State: Running, Role: LEADER
I20260619 09:04:48.028776 18973 consensus_queue.cc:237] T 192827240aa34527bb4e1c523f8791a1 P 3e080f9e327649bb98ff6d4abf29bf15 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e080f9e327649bb98ff6d4abf29bf15" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45955 } } peers { permanent_uuid: "292e54f4d4fa478ca0b060c275025ece" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34851 } } peers { permanent_uuid: "2cbc3aa2144f4f3181a0ca9553729cc9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40369 } }
I20260619 09:04:48.029610 18492 catalog_manager.cc:5495] T 192827240aa34527bb4e1c523f8791a1 P 3e080f9e327649bb98ff6d4abf29bf15 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3e080f9e327649bb98ff6d4abf29bf15 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "3e080f9e327649bb98ff6d4abf29bf15" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e080f9e327649bb98ff6d4abf29bf15" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45955 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "292e54f4d4fa478ca0b060c275025ece" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34851 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2cbc3aa2144f4f3181a0ca9553729cc9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40369 } health_report { overall_health: UNKNOWN } } }
I20260619 09:04:48.103654 18965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:04:48.101465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58732 (local address 127.0.98.2:45955)
0619 09:04:48.101532 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0619 09:04:48.101534 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:04:48.101541 (+ 7us) server_negotiation.cc:407] Connection header received
0619 09:04:48.101573 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:04:48.101573 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:04:48.101592 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:04:48.101649 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 09:04:48.101771 (+ 122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:04:48.102232 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:04:48.102966 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:04:48.103147 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:04:48.103200 (+ 53us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:04:48.103332 (+ 132us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 09:04:48.103421 (+ 89us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 09:04:48.103507 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0619 09:04:48.103560 (+ 53us) server_negotiation.cc:299] Negotiation successful
0619 09:04:48.103570 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260619 09:04:48.105309 18906 raft_consensus.cc:1261] T 192827240aa34527bb4e1c523f8791a1 P 2cbc3aa2144f4f3181a0ca9553729cc9 [term 1 FOLLOWER]: Refusing update from remote peer 3e080f9e327649bb98ff6d4abf29bf15: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 09:04:48.105309 18638 raft_consensus.cc:1261] T 192827240aa34527bb4e1c523f8791a1 P 292e54f4d4fa478ca0b060c275025ece [term 1 FOLLOWER]: Refusing update from remote peer 3e080f9e327649bb98ff6d4abf29bf15: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 09:04:48.105628 18973 consensus_queue.cc:1035] T 192827240aa34527bb4e1c523f8791a1 P 3e080f9e327649bb98ff6d4abf29bf15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2cbc3aa2144f4f3181a0ca9553729cc9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 09:04:48.105777 18976 consensus_queue.cc:1035] T 192827240aa34527bb4e1c523f8791a1 P 3e080f9e327649bb98ff6d4abf29bf15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "292e54f4d4fa478ca0b060c275025ece" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 09:04:48.108351 18980 mvcc.cc:205] Tried to move back new op lower bound from 7298498101676826624 to 7298498101367115776. Current Snapshot: MvccSnapshot[applied={T|T < 7298498101676826624}]
I20260619 09:04:48.109966 18493 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35422:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 09:04:48.110060 18493 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35422:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 09:04:48.110759 18493 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 23bf14b677cf4be68af716c334dac70e: Sending DeleteTablet for 3 replicas of tablet 192827240aa34527bb4e1c523f8791a1
I20260619 09:04:48.111080 18618 tablet_service.cc:1514] Processing DeleteTablet for tablet 192827240aa34527bb4e1c523f8791a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 09:04:48 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60076
I20260619 09:04:48.111135 18752 tablet_service.cc:1514] Processing DeleteTablet for tablet 192827240aa34527bb4e1c523f8791a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 09:04:48 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58722
I20260619 09:04:48.111292 18886 tablet_service.cc:1514] Processing DeleteTablet for tablet 192827240aa34527bb4e1c523f8791a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 09:04:48 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47856
I20260619 09:04:48.111493 18988 tablet_replica.cc:330] T 192827240aa34527bb4e1c523f8791a1 P 292e54f4d4fa478ca0b060c275025ece: stopping tablet replica
I20260619 09:04:48.111492 18990 tablet_replica.cc:330] T 192827240aa34527bb4e1c523f8791a1 P 2cbc3aa2144f4f3181a0ca9553729cc9: stopping tablet replica
I20260619 09:04:48.111505 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ez5NB/build/release/bin/kudu with pid 18555
I20260619 09:04:48.111585 18990 raft_consensus.cc:2229] T 192827240aa34527bb4e1c523f8791a1 P 2cbc3aa2144f4f3181a0ca9553729cc9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 09:04:48.111585 18988 raft_consensus.cc:2229] T 192827240aa34527bb4e1c523f8791a1 P 292e54f4d4fa478ca0b060c275025ece [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 09:04:48.111676 18990 raft_consensus.cc:2258] T 192827240aa34527bb4e1c523f8791a1 P 2cbc3aa2144f4f3181a0ca9553729cc9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 09:04:48.111676 18988 raft_consensus.cc:2258] T 192827240aa34527bb4e1c523f8791a1 P 292e54f4d4fa478ca0b060c275025ece [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 09:04:48.111816 18989 tablet_replica.cc:330] T 192827240aa34527bb4e1c523f8791a1 P 3e080f9e327649bb98ff6d4abf29bf15: stopping tablet replica
I20260619 09:04:48.111896 18989 raft_consensus.cc:2229] T 192827240aa34527bb4e1c523f8791a1 P 3e080f9e327649bb98ff6d4abf29bf15 [term 1 LEADER]: Raft consensus shutting down.
I20260619 09:04:48.111963 18989 raft_consensus.cc:2258] T 192827240aa34527bb4e1c523f8791a1 P 3e080f9e327649bb98ff6d4abf29bf15 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 09:04:48.112286 18990 ts_tablet_manager.cc:1905] T 192827240aa34527bb4e1c523f8791a1 P 2cbc3aa2144f4f3181a0ca9553729cc9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 09:04:48.113799 18989 ts_tablet_manager.cc:1905] T 192827240aa34527bb4e1c523f8791a1 P 3e080f9e327649bb98ff6d4abf29bf15: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 09:04:48.114933 18989 ts_tablet_manager.cc:1918] T 192827240aa34527bb4e1c523f8791a1 P 3e080f9e327649bb98ff6d4abf29bf15: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 09:04:48.114979 18989 log.cc:1196] T 192827240aa34527bb4e1c523f8791a1 P 3e080f9e327649bb98ff6d4abf29bf15: Deleting WAL directory at /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781859658134510-392-0/minicluster-data/ts-1/wal/wals/192827240aa34527bb4e1c523f8791a1
I20260619 09:04:48.115130 18989 ts_tablet_manager.cc:1939] T 192827240aa34527bb4e1c523f8791a1 P 3e080f9e327649bb98ff6d4abf29bf15: Deleting consensus metadata
I20260619 09:04:48.115285 18990 ts_tablet_manager.cc:1918] T 192827240aa34527bb4e1c523f8791a1 P 2cbc3aa2144f4f3181a0ca9553729cc9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 09:04:48.115339 18990 log.cc:1196] T 192827240aa34527bb4e1c523f8791a1 P 2cbc3aa2144f4f3181a0ca9553729cc9: Deleting WAL directory at /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781859658134510-392-0/minicluster-data/ts-2/wal/wals/192827240aa34527bb4e1c523f8791a1
I20260619 09:04:48.115573 18990 ts_tablet_manager.cc:1939] T 192827240aa34527bb4e1c523f8791a1 P 2cbc3aa2144f4f3181a0ca9553729cc9: Deleting consensus metadata
I20260619 09:04:48.115808 18478 catalog_manager.cc:4841] TS 3e080f9e327649bb98ff6d4abf29bf15 (127.0.98.2:45955): tablet 192827240aa34527bb4e1c523f8791a1 (table test-table [id=66f74f4fcb614a14bac62ab9b667f3a7]) successfully deleted
I20260619 09:04:48.115960 18478 catalog_manager.cc:4841] TS 2cbc3aa2144f4f3181a0ca9553729cc9 (127.0.98.3:40369): tablet 192827240aa34527bb4e1c523f8791a1 (table test-table [id=66f74f4fcb614a14bac62ab9b667f3a7]) successfully deleted
W20260619 09:04:48.119539 18477 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:34851 (error 108)
I20260619 09:04:48.119766 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ez5NB/build/release/bin/kudu with pid 18689
I20260619 09:04:48.120323 18963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:04:48.119843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34851 (local address 127.0.0.1:60090)
0619 09:04:48.120257 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0619 09:04:48.120292 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34851: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":396}
W20260619 09:04:48.120486 18477 catalog_manager.cc:4584] TS 292e54f4d4fa478ca0b060c275025ece (127.0.98.1:34851): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 192827240aa34527bb4e1c523f8791a1: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34851: connect: Connection refused (error 111)
I20260619 09:04:48.123991 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ez5NB/build/release/bin/kudu with pid 18823
I20260619 09:04:48.127794 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ez5NB/build/release/bin/kudu with pid 18461
2026-06-19T09:04:48Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2087 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-06-19T09:04:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T09:04:48Z Disabled control of system clock
I20260619 09:04:48.152374 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ez5NB/build/release/bin/kudu
/tmp/dist-test-task4ez5NB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781859658134510-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781859658134510-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781859658134510-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781859658134510-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46403
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:37821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46403
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 09:04:48.229056 18995 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 09:04:48.229254 18995 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 09:04:48.229274 18995 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 09:04:48.230341 18995 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 09:04:48.230372 18995 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 09:04:48.230384 18995 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 09:04:48.230394 18995 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 09:04:48.230405 18995 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 09:04:48.231664 18995 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781859658134510-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781859658134510-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46403
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46403
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781859658134510-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18995
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781859658134510-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 09:04:48.231839 18995 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 09:04:48.232046 18995 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 09:04:48.236560 18995 server_base.cc:1034] running on GCE node
W20260619 09:04:48.236521 19001 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:04:48.236528 19003 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:04:48.236537 19000 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:04:48.237013 18995 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 09:04:48.237270 18995 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 09:04:48.238422 18995 hybrid_clock.cc:648] HybridClock initialized: now 1781859888238405 us; error 26 us; skew 500 ppm
I20260619 09:04:48.239727 18995 webserver.cc:460] Webserver started at http://127.0.98.62:45611/ using document root <none> and password file <none>
I20260619 09:04:48.239949 18995 fs_manager.cc:359] Metadata directory not provided
I20260619 09:04:48.239998 18995 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 09:04:48.240134 18995 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 09:04:48.241130 18995 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781859658134510-392-0/minicluster-data/master-0/data/instance:
uuid: "5a180c40cf06444db06a290ede78e632"
format_stamp: "Formatted at 2026-06-19 09:04:48 on dist-test-slave-mtdr"
server_key: "d592ce7ea7e15886e74d679f46b5da26"
server_key_iv: "2a75a55140f6f98fd17d9be050a4059b"
server_key_version: "encryptionkey@0"
I20260619 09:04:48.241449 18995 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781859658134510-392-0/minicluster-data/master-0/wal/instance:
uuid: "5a180c40cf06444db06a290ede78e632"
format_stamp: "Formatted at 2026-06-19 09:04:48 on dist-test-slave-mtdr"
server_key: "d592ce7ea7e15886e74d679f46b5da26"
server_key_iv: "2a75a55140f6f98fd17d9be050a4059b"
server_key_version: "encryptionkey@0"
I20260619 09:04:48.242605 18995 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 09:04:48.243278 19009 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 09:04:48.243436 18995 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260619 09:04:48.243515 18995 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781859658134510-392-0/minicluster-data/master-0/data,/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781859658134510-392-0/minicluster-data/master-0/wal
uuid: "5a180c40cf06444db06a290ede78e632"
format_stamp: "Formatted at 2026-06-19 09:04:48 on dist-test-slave-mtdr"
server_key: "d592ce7ea7e15886e74d679f46b5da26"
server_key_iv: "2a75a55140f6f98fd17d9be050a4059b"
server_key_version: "encryptionkey@0"
I20260619 09:04:48.243577 18995 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781859658134510-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781859658134510-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781859658134510-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 09:04:48.252518 18995 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 09:04:48.252761 18995 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 09:04:48.252873 18995 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 09:04:48.256493 18995 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46403
I20260619 09:04:48.256528 19061 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46403 every 8 connection(s)
I20260619 09:04:48.256809 18995 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781859658134510-392-0/minicluster-data/master-0/data/info.pb
I20260619 09:04:48.257675 19062 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 09:04:48.260027 19062 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a180c40cf06444db06a290ede78e632: Bootstrap starting.
I20260619 09:04:48.260572 19062 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5a180c40cf06444db06a290ede78e632: Neither blocks nor log segments found. Creating new log.
I20260619 09:04:48.260910 19062 log.cc:824] T 00000000000000000000000000000000 P 5a180c40cf06444db06a290ede78e632: Log is configured to *not* fsync() on all Append() calls
I20260619 09:04:48.261566 19062 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a180c40cf06444db06a290ede78e632: No bootstrap required, opened a new log
I20260619 09:04:48.263197 19062 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5a180c40cf06444db06a290ede78e632 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a180c40cf06444db06a290ede78e632" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46403 } }
I20260619 09:04:48.263336 19062 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5a180c40cf06444db06a290ede78e632 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 09:04:48.263361 19062 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5a180c40cf06444db06a290ede78e632 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a180c40cf06444db06a290ede78e632, State: Initialized, Role: FOLLOWER
I20260619 09:04:48.263474 19062 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5a180c40cf06444db06a290ede78e632 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a180c40cf06444db06a290ede78e632" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46403 } }
I20260619 09:04:48.263552 19062 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5a180c40cf06444db06a290ede78e632 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 09:04:48.263586 19062 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5a180c40cf06444db06a290ede78e632 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 09:04:48.263620 19062 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5a180c40cf06444db06a290ede78e632 [term 0 FOLLOWER]: Advancing to term 1
I20260619 09:04:48.264084 19062 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5a180c40cf06444db06a290ede78e632 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a180c40cf06444db06a290ede78e632" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46403 } }
I20260619 09:04:48.264169 19062 leader_election.cc:302] T 00000000000000000000000000000000 P 5a180c40cf06444db06a290ede78e632 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5a180c40cf06444db06a290ede78e632; no voters:
I20260619 09:04:48.264297 19062 leader_election.cc:288] T 00000000000000000000000000000000 P 5a180c40cf06444db06a290ede78e632 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 09:04:48.264353 19065 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5a180c40cf06444db06a290ede78e632 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 09:04:48.264498 19065 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5a180c40cf06444db06a290ede78e632 [term 1 LEADER]: Becoming Leader. State: Replica: 5a180c40cf06444db06a290ede78e632, State: Running, Role: LEADER
I20260619 09:04:48.264532 19062 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5a180c40cf06444db06a290ede78e632 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 09:04:48.264595 19065 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5a180c40cf06444db06a290ede78e632 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a180c40cf06444db06a290ede78e632" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46403 } }
I20260619 09:04:48.264978 19067 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a180c40cf06444db06a290ede78e632 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5a180c40cf06444db06a290ede78e632. Latest consensus state: current_term: 1 leader_uuid: "5a180c40cf06444db06a290ede78e632" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a180c40cf06444db06a290ede78e632" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46403 } } }
I20260619 09:04:48.265070 19067 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a180c40cf06444db06a290ede78e632 [sys.catalog]: This master's current role is: LEADER
I20260619 09:04:48.265040 19066 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a180c40cf06444db06a290ede78e632 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5a180c40cf06444db06a290ede78e632" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a180c40cf06444db06a290ede78e632" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46403 } } }
I20260619 09:04:48.265125 19066 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a180c40cf06444db06a290ede78e632 [sys.catalog]: This master's current role is: LEADER
I20260619 09:04:48.265290 19074 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 09:04:48.265780 19074 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 09:04:48.265875 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ez5NB/build/release/bin/kudu as pid 18995
I20260619 09:04:48.265967 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781859658134510-392-0/minicluster-data/master-0/wal/instance
I20260619 09:04:48.266211 392 external_mini_cluster.cc:1428] Setting key ffb8e4548dcb72accd674db56c9ff00c
I20260619 09:04:48.267292 19074 catalog_manager.cc:1347] Generated new cluster ID: 7ee2e0b2aa40446d8a23bdff4f7298c2
I20260619 09:04:48.267346 19074 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 09:04:48.271075 19084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:04:48.266659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34536 (local address 127.0.98.62:46403)
0619 09:04:48.266980 (+ 321us) server_negotiation.cc:206] Beginning negotiation
0619 09:04:48.266988 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 09:04:48.267017 (+ 29us) server_negotiation.cc:407] Connection header received
0619 09:04:48.267131 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:04:48.267139 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:04:48.267205 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:04:48.267363 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:04:48.267594 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:04:48.268489 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:04:48.269243 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:04:48.269651 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:04:48.269753 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:04:48.270331 (+ 578us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:04:48.270335 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:04:48.270336 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 09:04:48.270495 (+ 159us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:04:48.270527 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 09:04:48.270538 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:04:48.270590 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0619 09:04:48.270686 (+ 96us) server_negotiation.cc:299] Negotiation successful
0619 09:04:48.270880 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":256,"thread_start_us":71,"threads_started":1}
I20260619 09:04:48.301158 19074 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 09:04:48.301893 19074 catalog_manager.cc:1504] Loading token signing keys...
I20260619 09:04:48.322678 19074 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5a180c40cf06444db06a290ede78e632: Generated new TSK 0
I20260619 09:04:48.322916 19074 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 09:04:48.328397 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ez5NB/build/release/bin/kudu
/tmp/dist-test-task4ez5NB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781859658134510-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781859658134510-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781859658134510-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781859658134510-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46403
--builtin_ntp_servers=127.0.98.20:37821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 09:04:48.409219 19086 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 09:04:48.409410 19086 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 09:04:48.409447 19086 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 09:04:48.410557 19086 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 09:04:48.410593 19086 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 09:04:48.410624 19086 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 09:04:48.411972 19086 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781859658134510-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781859658134510-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781859658134510-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46403
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19086
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781859658134510-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 09:04:48.412205 19086 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 09:04:48.412482 19086 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 09:04:48.417183 19091 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:04:48.417179 19094 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:04:48.417179 19092 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:04:48.417214 19086 server_base.cc:1034] running on GCE node
I20260619 09:04:48.417596 19086 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 09:04:48.417812 19086 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 09:04:48.418941 19086 hybrid_clock.cc:648] HybridClock initialized: now 1781859888418924 us; error 32 us; skew 500 ppm
I20260619 09:04:48.420142 19086 webserver.cc:460] Webserver started at http://127.0.98.1:34935/ using document root <none> and password file <none>
I20260619 09:04:48.420375 19086 fs_manager.cc:359] Metadata directory not provided
I20260619 09:04:48.420423 19086 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 09:04:48.420527 19086 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 09:04:48.421532 19086 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781859658134510-392-0/minicluster-data/ts-0/data/instance:
uuid: "6aace6fb21ae40ce8d02a88f873ddbb6"
format_stamp: "Formatted at 2026-06-19 09:04:48 on dist-test-slave-mtdr"
server_key: "2e944e662485d24722f6bf80458db801"
server_key_iv: "3595966da613799bd39071f3712c6048"
server_key_version: "encryptionkey@0"
I20260619 09:04:48.421844 19086 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781859658134510-392-0/minicluster-data/ts-0/wal/instance:
uuid: "6aace6fb21ae40ce8d02a88f873ddbb6"
format_stamp: "Formatted at 2026-06-19 09:04:48 on dist-test-slave-mtdr"
server_key: "2e944e662485d24722f6bf80458db801"
server_key_iv: "3595966da613799bd39071f3712c6048"
server_key_version: "encryptionkey@0"
I20260619 09:04:48.423072 19086 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260619 09:04:48.423772 19100 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 09:04:48.423919 19086 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260619 09:04:48.423990 19086 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781859658134510-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781859658134510-392-0/minicluster-data/ts-0/wal
uuid: "6aace6fb21ae40ce8d02a88f873ddbb6"
format_stamp: "Formatted at 2026-06-19 09:04:48 on dist-test-slave-mtdr"
server_key: "2e944e662485d24722f6bf80458db801"
server_key_iv: "3595966da613799bd39071f3712c6048"
server_key_version: "encryptionkey@0"
I20260619 09:04:48.424052 19086 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781859658134510-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781859658134510-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781859658134510-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 09:04:48.437717 19086 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 09:04:48.437994 19086 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 09:04:48.438105 19086 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 09:04:48.438303 19086 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 09:04:48.438609 19086 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 09:04:48.438643 19086 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 09:04:48.438670 19086 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 09:04:48.438686 19086 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 09:04:48.443980 19086 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44071
I20260619 09:04:48.444044 19213 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44071 every 8 connection(s)
I20260619 09:04:48.444317 19086 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781859658134510-392-0/minicluster-data/ts-0/data/info.pb
I20260619 09:04:48.449167 19084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:04:48.445002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42599 (local address 127.0.98.62:46403)
0619 09:04:48.445090 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0619 09:04:48.445092 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:04:48.445251 (+ 159us) server_negotiation.cc:407] Connection header received
0619 09:04:48.445722 (+ 471us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:04:48.445724 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:04:48.445738 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:04:48.445803 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:04:48.446534 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:04:48.447112 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:04:48.448049 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:04:48.448246 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:04:48.448295 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:04:48.448966 (+ 671us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:04:48.448967 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:04:48.448967 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 09:04:48.448971 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:04:48.448986 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 09:04:48.448990 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:04:48.449028 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0619 09:04:48.449087 (+ 59us) server_negotiation.cc:299] Negotiation successful
0619 09:04:48.449108 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 09:04:48.449332 19216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:04:48.444949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46403 (local address 127.0.98.1:42599)
0619 09:04:48.445117 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0619 09:04:48.445156 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 09:04:48.445619 (+ 463us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:04:48.445866 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:04:48.445869 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:04:48.445876 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0619 09:04:48.446441 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:04:48.446446 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:04:48.447183 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:04:48.447183 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:04:48.447985 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:04:48.447988 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:04:48.448303 (+ 315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:04:48.448304 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:04:48.448419 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:04:48.448852 (+ 433us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 09:04:48.448852 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0619 09:04:48.448880 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 09:04:48.448901 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 09:04:48.449034 (+ 133us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 09:04:48.449036 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 09:04:48.449038 (+ 2us) client_negotiation.cc:769] Sending connection context
0619 09:04:48.449083 (+ 45us) client_negotiation.cc:240] Negotiation successful
0619 09:04:48.449154 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":65,"threads_started":1}
I20260619 09:04:48.449746 19214 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46403
I20260619 09:04:48.449823 19214 heartbeater.cc:464] Registering TS with master...
I20260619 09:04:48.449980 19214 heartbeater.cc:511] Master 127.0.98.62:46403 requested a full tablet report, sending...
I20260619 09:04:48.450425 19026 ts_manager.cc:194] Registered new tserver with Master: 6aace6fb21ae40ce8d02a88f873ddbb6 (127.0.98.1:44071)
I20260619 09:04:48.451042 19026 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:42599
I20260619 09:04:48.453625 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ez5NB/build/release/bin/kudu as pid 19086
I20260619 09:04:48.453728 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781859658134510-392-0/minicluster-data/ts-0/wal/instance
I20260619 09:04:48.453941 392 external_mini_cluster.cc:1428] Setting key 04be644c0eaff86d08dc95aa6fa7922b
I20260619 09:04:48.455078 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ez5NB/build/release/bin/kudu
/tmp/dist-test-task4ez5NB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781859658134510-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781859658134510-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781859658134510-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781859658134510-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46403
--builtin_ntp_servers=127.0.98.20:37821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 09:04:48.531920 19217 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 09:04:48.532111 19217 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 09:04:48.532138 19217 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 09:04:48.533237 19217 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 09:04:48.533277 19217 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 09:04:48.533315 19217 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 09:04:48.534588 19217 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781859658134510-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781859658134510-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781859658134510-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46403
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19217
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781859658134510-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 09:04:48.534802 19217 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 09:04:48.535071 19217 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 09:04:48.539701 19223 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:04:48.539724 19222 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:04:48.539788 19225 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:04:48.539974 19217 server_base.cc:1034] running on GCE node
I20260619 09:04:48.540237 19217 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 09:04:48.540477 19217 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 09:04:48.541666 19217 hybrid_clock.cc:648] HybridClock initialized: now 1781859888541648 us; error 31 us; skew 500 ppm
I20260619 09:04:48.542884 19217 webserver.cc:460] Webserver started at http://127.0.98.2:41735/ using document root <none> and password file <none>
I20260619 09:04:48.543093 19217 fs_manager.cc:359] Metadata directory not provided
I20260619 09:04:48.543147 19217 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 09:04:48.543305 19217 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 09:04:48.544341 19217 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781859658134510-392-0/minicluster-data/ts-1/data/instance:
uuid: "7379e7e0ad2b4ac08f40c3fe7e916f32"
format_stamp: "Formatted at 2026-06-19 09:04:48 on dist-test-slave-mtdr"
server_key: "d8aee29bbd7e43eb665f22f31bd417a0"
server_key_iv: "ee6ae0173cfeb621fa06c447cb455844"
server_key_version: "encryptionkey@0"
I20260619 09:04:48.544652 19217 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781859658134510-392-0/minicluster-data/ts-1/wal/instance:
uuid: "7379e7e0ad2b4ac08f40c3fe7e916f32"
format_stamp: "Formatted at 2026-06-19 09:04:48 on dist-test-slave-mtdr"
server_key: "d8aee29bbd7e43eb665f22f31bd417a0"
server_key_iv: "ee6ae0173cfeb621fa06c447cb455844"
server_key_version: "encryptionkey@0"
I20260619 09:04:48.545830 19217 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 09:04:48.546535 19231 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 09:04:48.546675 19217 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 09:04:48.546754 19217 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781859658134510-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781859658134510-392-0/minicluster-data/ts-1/wal
uuid: "7379e7e0ad2b4ac08f40c3fe7e916f32"
format_stamp: "Formatted at 2026-06-19 09:04:48 on dist-test-slave-mtdr"
server_key: "d8aee29bbd7e43eb665f22f31bd417a0"
server_key_iv: "ee6ae0173cfeb621fa06c447cb455844"
server_key_version: "encryptionkey@0"
I20260619 09:04:48.546842 19217 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781859658134510-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781859658134510-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781859658134510-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 09:04:48.569582 19217 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 09:04:48.569875 19217 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 09:04:48.569998 19217 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 09:04:48.570216 19217 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 09:04:48.570528 19217 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 09:04:48.570561 19217 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 09:04:48.570590 19217 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 09:04:48.570614 19217 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 09:04:48.575824 19217 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34195
I20260619 09:04:48.575865 19344 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34195 every 8 connection(s)
I20260619 09:04:48.576167 19217 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781859658134510-392-0/minicluster-data/ts-1/data/info.pb
I20260619 09:04:48.579509 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ez5NB/build/release/bin/kudu as pid 19217
I20260619 09:04:48.579622 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781859658134510-392-0/minicluster-data/ts-1/wal/instance
I20260619 09:04:48.579847 392 external_mini_cluster.cc:1428] Setting key f284c8b1975469c14c7508d931fe3d8a
I20260619 09:04:48.580802 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ez5NB/build/release/bin/kudu
/tmp/dist-test-task4ez5NB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781859658134510-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781859658134510-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781859658134510-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781859658134510-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46403
--builtin_ntp_servers=127.0.98.20:37821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 09:04:48.581800 19084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:04:48.576915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:32879 (local address 127.0.98.62:46403)
0619 09:04:48.576972 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0619 09:04:48.576974 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:04:48.577200 (+ 226us) server_negotiation.cc:407] Connection header received
0619 09:04:48.577673 (+ 473us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:04:48.577675 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:04:48.577686 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:04:48.577747 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:04:48.578449 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:04:48.579001 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:04:48.580345 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:04:48.580621 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:04:48.580674 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:04:48.581565 (+ 891us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:04:48.581566 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:04:48.581566 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 09:04:48.581570 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:04:48.581584 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 09:04:48.581588 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:04:48.581631 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0619 09:04:48.581719 (+ 88us) server_negotiation.cc:299] Negotiation successful
0619 09:04:48.581742 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260619 09:04:48.582055 19347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:04:48.576865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46403 (local address 127.0.98.2:32879)
0619 09:04:48.577045 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0619 09:04:48.577086 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 09:04:48.577546 (+ 460us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:04:48.577801 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:04:48.577804 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:04:48.577812 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0619 09:04:48.578288 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:04:48.578294 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:04:48.579102 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:04:48.579103 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:04:48.580259 (+ 1156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:04:48.580263 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:04:48.580702 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:04:48.580704 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:04:48.580861 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:04:48.581411 (+ 550us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 09:04:48.581412 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 09:04:48.581451 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 09:04:48.581480 (+ 29us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 09:04:48.581649 (+ 169us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 09:04:48.581652 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 09:04:48.581655 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 09:04:48.581718 (+ 63us) client_negotiation.cc:240] Negotiation successful
0619 09:04:48.581836 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":64,"threads_started":1}
I20260619 09:04:48.582636 19345 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46403
I20260619 09:04:48.582741 19345 heartbeater.cc:464] Registering TS with master...
I20260619 09:04:48.582959 19345 heartbeater.cc:511] Master 127.0.98.62:46403 requested a full tablet report, sending...
I20260619 09:04:48.583385 19026 ts_manager.cc:194] Registered new tserver with Master: 7379e7e0ad2b4ac08f40c3fe7e916f32 (127.0.98.2:34195)
I20260619 09:04:48.583841 19026 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:32879
W20260619 09:04:48.658839 19348 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 09:04:48.659018 19348 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 09:04:48.659039 19348 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 09:04:48.660140 19348 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 09:04:48.660179 19348 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 09:04:48.660202 19348 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 09:04:48.661482 19348 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781859658134510-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781859658134510-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781859658134510-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46403
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19348
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781859658134510-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 09:04:48.661731 19348 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 09:04:48.662006 19348 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 09:04:48.666801 19353 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:04:48.666859 19354 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:04:48.666796 19356 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:04:48.666935 19348 server_base.cc:1034] running on GCE node
I20260619 09:04:48.667210 19348 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 09:04:48.667455 19348 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 09:04:48.668639 19348 hybrid_clock.cc:648] HybridClock initialized: now 1781859888668621 us; error 31 us; skew 500 ppm
I20260619 09:04:48.669837 19348 webserver.cc:460] Webserver started at http://127.0.98.3:36901/ using document root <none> and password file <none>
I20260619 09:04:48.670053 19348 fs_manager.cc:359] Metadata directory not provided
I20260619 09:04:48.670109 19348 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 09:04:48.670239 19348 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 09:04:48.671304 19348 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781859658134510-392-0/minicluster-data/ts-2/data/instance:
uuid: "f878dce091dc417fa027e5a300f44517"
format_stamp: "Formatted at 2026-06-19 09:04:48 on dist-test-slave-mtdr"
server_key: "45c133ae205385e13f3f9ded6d54a542"
server_key_iv: "1430b6636809ce4754be53b3e3ba6773"
server_key_version: "encryptionkey@0"
I20260619 09:04:48.671651 19348 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781859658134510-392-0/minicluster-data/ts-2/wal/instance:
uuid: "f878dce091dc417fa027e5a300f44517"
format_stamp: "Formatted at 2026-06-19 09:04:48 on dist-test-slave-mtdr"
server_key: "45c133ae205385e13f3f9ded6d54a542"
server_key_iv: "1430b6636809ce4754be53b3e3ba6773"
server_key_version: "encryptionkey@0"
I20260619 09:04:48.672950 19348 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 09:04:48.673724 19362 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 09:04:48.673887 19348 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 09:04:48.673976 19348 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781859658134510-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781859658134510-392-0/minicluster-data/ts-2/wal
uuid: "f878dce091dc417fa027e5a300f44517"
format_stamp: "Formatted at 2026-06-19 09:04:48 on dist-test-slave-mtdr"
server_key: "45c133ae205385e13f3f9ded6d54a542"
server_key_iv: "1430b6636809ce4754be53b3e3ba6773"
server_key_version: "encryptionkey@0"
I20260619 09:04:48.674029 19348 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781859658134510-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781859658134510-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781859658134510-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 09:04:48.718761 19348 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 09:04:48.719108 19348 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 09:04:48.719223 19348 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 09:04:48.719436 19348 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 09:04:48.719762 19348 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 09:04:48.719797 19348 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 09:04:48.719828 19348 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 09:04:48.719846 19348 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 09:04:48.725549 19348 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34721
I20260619 09:04:48.725589 19475 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34721 every 8 connection(s)
I20260619 09:04:48.725934 19348 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781859658134510-392-0/minicluster-data/ts-2/data/info.pb
I20260619 09:04:48.726323 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ez5NB/build/release/bin/kudu as pid 19348
I20260619 09:04:48.726428 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781859658134510-392-0/minicluster-data/ts-2/wal/instance
I20260619 09:04:48.726615 392 external_mini_cluster.cc:1428] Setting key 6feb19840a79afcb1515b7c7477e8f68
I20260619 09:04:48.731444 19084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:04:48.726695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60125 (local address 127.0.98.62:46403)
0619 09:04:48.726747 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0619 09:04:48.726750 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 09:04:48.727079 (+ 329us) server_negotiation.cc:407] Connection header received
0619 09:04:48.727725 (+ 646us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:04:48.727725 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:04:48.727735 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:04:48.727782 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:04:48.728456 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:04:48.728970 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:04:48.729973 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:04:48.730173 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:04:48.730255 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:04:48.731152 (+ 897us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:04:48.731153 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:04:48.731154 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 09:04:48.731158 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:04:48.731174 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 09:04:48.731178 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:04:48.731230 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0619 09:04:48.731349 (+ 119us) server_negotiation.cc:299] Negotiation successful
0619 09:04:48.731378 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260619 09:04:48.731678 19478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:04:48.726679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46403 (local address 127.0.98.3:60125)
0619 09:04:48.726958 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0619 09:04:48.727004 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 09:04:48.727635 (+ 631us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:04:48.727859 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:04:48.727862 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:04:48.727870 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0619 09:04:48.728361 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:04:48.728365 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:04:48.729065 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:04:48.729065 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:04:48.729887 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:04:48.729889 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:04:48.730304 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:04:48.730305 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:04:48.730421 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:04:48.731003 (+ 582us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 09:04:48.731004 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 09:04:48.731033 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 09:04:48.731057 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 09:04:48.731234 (+ 177us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 09:04:48.731237 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 09:04:48.731240 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 09:04:48.731312 (+ 72us) client_negotiation.cc:240] Negotiation successful
0619 09:04:48.731395 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":131,"threads_started":1}
I20260619 09:04:48.732163 19476 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46403
I20260619 09:04:48.732250 19476 heartbeater.cc:464] Registering TS with master...
I20260619 09:04:48.732435 19476 heartbeater.cc:511] Master 127.0.98.62:46403 requested a full tablet report, sending...
I20260619 09:04:48.732824 19026 ts_manager.cc:194] Registered new tserver with Master: f878dce091dc417fa027e5a300f44517 (127.0.98.3:34721)
I20260619 09:04:48.733196 19026 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:60125
I20260619 09:04:48.738561 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 09:04:48.742899 19084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:04:48.740003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34548 (local address 127.0.98.62:46403)
0619 09:04:48.740064 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0619 09:04:48.740066 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:04:48.740128 (+ 62us) server_negotiation.cc:407] Connection header received
0619 09:04:48.740223 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:04:48.740224 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:04:48.740236 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:04:48.740283 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:04:48.740435 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:04:48.740924 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:04:48.741646 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:04:48.741849 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:04:48.741904 (+ 55us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:04:48.742574 (+ 670us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:04:48.742576 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:04:48.742576 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 09:04:48.742581 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:04:48.742597 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 09:04:48.742601 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:04:48.742697 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0619 09:04:48.742767 (+ 70us) server_negotiation.cc:299] Negotiation successful
0619 09:04:48.742778 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260619 09:04:48.744746 19026 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34548:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 09:04:48.746475 19026 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 09:04:48.752822 19488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:04:48.748880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49968 (local address 127.0.98.2:34195)
0619 09:04:48.749148 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0619 09:04:48.749153 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 09:04:48.749168 (+ 15us) server_negotiation.cc:407] Connection header received
0619 09:04:48.749208 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:04:48.749210 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:04:48.749226 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:04:48.749340 (+ 114us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 09:04:48.749628 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:04:48.750704 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:04:48.751971 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:04:48.752440 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:04:48.752505 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:04:48.752598 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0619 09:04:48.752685 (+ 87us) server_negotiation.cc:299] Negotiation successful
0619 09:04:48.752721 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":230,"thread_start_us":70,"threads_started":1}
I20260619 09:04:48.752888 19484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:04:48.748631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34721 (local address 127.0.0.1:39528)
0619 09:04:48.748798 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0619 09:04:48.748836 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 09:04:48.748903 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:04:48.749367 (+ 464us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:04:48.749371 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:04:48.749374 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 09:04:48.749449 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:04:48.749451 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:04:48.750491 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:04:48.750492 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:04:48.751728 (+ 1236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:04:48.751730 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:04:48.752634 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:04:48.752635 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:04:48.752694 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:04:48.752699 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 09:04:48.752728 (+ 29us) client_negotiation.cc:240] Negotiation successful
0619 09:04:48.752768 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":91,"threads_started":1}
I20260619 09:04:48.753055 19486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:04:48.748631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44071 (local address 127.0.0.1:46314)
0619 09:04:48.749003 (+ 372us) negotiation.cc:107] Waiting for socket to connect
0619 09:04:48.749011 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0619 09:04:48.749046 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:04:48.749765 (+ 719us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:04:48.749766 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:04:48.749768 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 09:04:48.749807 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:04:48.749809 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:04:48.751047 (+ 1238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:04:48.751048 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:04:48.751917 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:04:48.751920 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:04:48.752845 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:04:48.752846 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:04:48.752905 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:04:48.752908 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 09:04:48.752953 (+ 45us) client_negotiation.cc:240] Negotiation successful
0619 09:04:48.752991 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":361,"mutex_wait_us":3,"thread_start_us":58,"threads_started":1}
I20260619 09:04:48.753239 19489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:04:48.748984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39528 (local address 127.0.98.3:34721)
0619 09:04:48.749148 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0619 09:04:48.749153 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 09:04:48.749164 (+ 11us) server_negotiation.cc:407] Connection header received
0619 09:04:48.749208 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:04:48.749210 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:04:48.749226 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:04:48.749340 (+ 114us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 09:04:48.749521 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:04:48.750423 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:04:48.751820 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:04:48.752521 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:04:48.752871 (+ 350us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:04:48.753088 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0619 09:04:48.753114 (+ 26us) server_negotiation.cc:299] Negotiation successful
0619 09:04:48.753155 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":84,"threads_started":1}
I20260619 09:04:48.753283 19487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:04:48.748802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46314 (local address 127.0.98.1:44071)
0619 09:04:48.749460 (+ 658us) server_negotiation.cc:206] Beginning negotiation
0619 09:04:48.749464 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 09:04:48.749473 (+ 9us) server_negotiation.cc:407] Connection header received
0619 09:04:48.749508 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:04:48.749510 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:04:48.749524 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:04:48.749698 (+ 174us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 09:04:48.749892 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:04:48.750982 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:04:48.752004 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:04:48.752680 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:04:48.752736 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:04:48.752815 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0619 09:04:48.753169 (+ 354us) server_negotiation.cc:299] Negotiation successful
0619 09:04:48.753209 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":621,"thread_start_us":89,"threads_started":1}
I20260619 09:04:48.752830 19485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:04:48.748632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34195 (local address 127.0.0.1:49968)
0619 09:04:48.748870 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0619 09:04:48.748880 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 09:04:48.748929 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:04:48.749525 (+ 596us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:04:48.749525 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:04:48.749527 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 09:04:48.749557 (+ 30us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:04:48.749558 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:04:48.750784 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:04:48.750785 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:04:48.751906 (+ 1121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:04:48.751908 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:04:48.752514 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:04:48.752515 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:04:48.752590 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:04:48.752603 (+ 13us) client_negotiation.cc:769] Sending connection context
0619 09:04:48.752672 (+ 69us) client_negotiation.cc:240] Negotiation successful
0619 09:04:48.752721 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":64,"threads_started":1}
I20260619 09:04:48.753952 19410 tablet_service.cc:1467] Processing CreateTablet for tablet babbe0a35ae94c7eb9ed547b1dfa4aba (DEFAULT_TABLE table=test-table [id=fd521b8b8b5c4af5b7ac4bff8ad4b977]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 09:04:48.754101 19279 tablet_service.cc:1467] Processing CreateTablet for tablet babbe0a35ae94c7eb9ed547b1dfa4aba (DEFAULT_TABLE table=test-table [id=fd521b8b8b5c4af5b7ac4bff8ad4b977]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 09:04:48.754365 19410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet babbe0a35ae94c7eb9ed547b1dfa4aba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 09:04:48.754398 19279 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet babbe0a35ae94c7eb9ed547b1dfa4aba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 09:04:48.755249 19148 tablet_service.cc:1467] Processing CreateTablet for tablet babbe0a35ae94c7eb9ed547b1dfa4aba (DEFAULT_TABLE table=test-table [id=fd521b8b8b5c4af5b7ac4bff8ad4b977]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 09:04:48.755594 19148 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet babbe0a35ae94c7eb9ed547b1dfa4aba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 09:04:48.757196 19490 tablet_bootstrap.cc:492] T babbe0a35ae94c7eb9ed547b1dfa4aba P 7379e7e0ad2b4ac08f40c3fe7e916f32: Bootstrap starting.
I20260619 09:04:48.757752 19490 tablet_bootstrap.cc:654] T babbe0a35ae94c7eb9ed547b1dfa4aba P 7379e7e0ad2b4ac08f40c3fe7e916f32: Neither blocks nor log segments found. Creating new log.
I20260619 09:04:48.757833 19491 tablet_bootstrap.cc:492] T babbe0a35ae94c7eb9ed547b1dfa4aba P f878dce091dc417fa027e5a300f44517: Bootstrap starting.
I20260619 09:04:48.758050 19490 log.cc:824] T babbe0a35ae94c7eb9ed547b1dfa4aba P 7379e7e0ad2b4ac08f40c3fe7e916f32: Log is configured to *not* fsync() on all Append() calls
I20260619 09:04:48.758433 19491 tablet_bootstrap.cc:654] T babbe0a35ae94c7eb9ed547b1dfa4aba P f878dce091dc417fa027e5a300f44517: Neither blocks nor log segments found. Creating new log.
I20260619 09:04:48.758697 19491 log.cc:824] T babbe0a35ae94c7eb9ed547b1dfa4aba P f878dce091dc417fa027e5a300f44517: Log is configured to *not* fsync() on all Append() calls
I20260619 09:04:48.758984 19490 tablet_bootstrap.cc:492] T babbe0a35ae94c7eb9ed547b1dfa4aba P 7379e7e0ad2b4ac08f40c3fe7e916f32: No bootstrap required, opened a new log
I20260619 09:04:48.759057 19490 ts_tablet_manager.cc:1397] T babbe0a35ae94c7eb9ed547b1dfa4aba P 7379e7e0ad2b4ac08f40c3fe7e916f32: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 09:04:48.759178 19493 tablet_bootstrap.cc:492] T babbe0a35ae94c7eb9ed547b1dfa4aba P 6aace6fb21ae40ce8d02a88f873ddbb6: Bootstrap starting.
I20260619 09:04:48.759430 19491 tablet_bootstrap.cc:492] T babbe0a35ae94c7eb9ed547b1dfa4aba P f878dce091dc417fa027e5a300f44517: No bootstrap required, opened a new log
I20260619 09:04:48.759501 19491 ts_tablet_manager.cc:1397] T babbe0a35ae94c7eb9ed547b1dfa4aba P f878dce091dc417fa027e5a300f44517: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 09:04:48.759790 19493 tablet_bootstrap.cc:654] T babbe0a35ae94c7eb9ed547b1dfa4aba P 6aace6fb21ae40ce8d02a88f873ddbb6: Neither blocks nor log segments found. Creating new log.
I20260619 09:04:48.760102 19493 log.cc:824] T babbe0a35ae94c7eb9ed547b1dfa4aba P 6aace6fb21ae40ce8d02a88f873ddbb6: Log is configured to *not* fsync() on all Append() calls
I20260619 09:04:48.760965 19493 tablet_bootstrap.cc:492] T babbe0a35ae94c7eb9ed547b1dfa4aba P 6aace6fb21ae40ce8d02a88f873ddbb6: No bootstrap required, opened a new log
I20260619 09:04:48.761029 19493 ts_tablet_manager.cc:1397] T babbe0a35ae94c7eb9ed547b1dfa4aba P 6aace6fb21ae40ce8d02a88f873ddbb6: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260619 09:04:48.761343 19490 raft_consensus.cc:348] T babbe0a35ae94c7eb9ed547b1dfa4aba P 7379e7e0ad2b4ac08f40c3fe7e916f32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f878dce091dc417fa027e5a300f44517" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34721 } } peers { permanent_uuid: "6aace6fb21ae40ce8d02a88f873ddbb6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44071 } } peers { permanent_uuid: "7379e7e0ad2b4ac08f40c3fe7e916f32" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34195 } }
I20260619 09:04:48.761509 19490 raft_consensus.cc:374] T babbe0a35ae94c7eb9ed547b1dfa4aba P 7379e7e0ad2b4ac08f40c3fe7e916f32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 09:04:48.761550 19490 raft_consensus.cc:729] T babbe0a35ae94c7eb9ed547b1dfa4aba P 7379e7e0ad2b4ac08f40c3fe7e916f32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7379e7e0ad2b4ac08f40c3fe7e916f32, State: Initialized, Role: FOLLOWER
I20260619 09:04:48.761605 19491 raft_consensus.cc:348] T babbe0a35ae94c7eb9ed547b1dfa4aba P f878dce091dc417fa027e5a300f44517 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f878dce091dc417fa027e5a300f44517" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34721 } } peers { permanent_uuid: "6aace6fb21ae40ce8d02a88f873ddbb6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44071 } } peers { permanent_uuid: "7379e7e0ad2b4ac08f40c3fe7e916f32" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34195 } }
I20260619 09:04:48.761768 19491 raft_consensus.cc:374] T babbe0a35ae94c7eb9ed547b1dfa4aba P f878dce091dc417fa027e5a300f44517 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 09:04:48.761739 19490 consensus_queue.cc:260] T babbe0a35ae94c7eb9ed547b1dfa4aba P 7379e7e0ad2b4ac08f40c3fe7e916f32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f878dce091dc417fa027e5a300f44517" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34721 } } peers { permanent_uuid: "6aace6fb21ae40ce8d02a88f873ddbb6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44071 } } peers { permanent_uuid: "7379e7e0ad2b4ac08f40c3fe7e916f32" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34195 } }
I20260619 09:04:48.761806 19491 raft_consensus.cc:729] T babbe0a35ae94c7eb9ed547b1dfa4aba P f878dce091dc417fa027e5a300f44517 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f878dce091dc417fa027e5a300f44517, State: Initialized, Role: FOLLOWER
I20260619 09:04:48.761927 19491 consensus_queue.cc:260] T babbe0a35ae94c7eb9ed547b1dfa4aba P f878dce091dc417fa027e5a300f44517 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f878dce091dc417fa027e5a300f44517" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34721 } } peers { permanent_uuid: "6aace6fb21ae40ce8d02a88f873ddbb6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44071 } } peers { permanent_uuid: "7379e7e0ad2b4ac08f40c3fe7e916f32" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34195 } }
I20260619 09:04:48.761999 19490 ts_tablet_manager.cc:1428] T babbe0a35ae94c7eb9ed547b1dfa4aba P 7379e7e0ad2b4ac08f40c3fe7e916f32: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260619 09:04:48.762135 19345 heartbeater.cc:503] Master 127.0.98.62:46403 was elected leader, sending a full tablet report...
I20260619 09:04:48.762174 19491 ts_tablet_manager.cc:1428] T babbe0a35ae94c7eb9ed547b1dfa4aba P f878dce091dc417fa027e5a300f44517: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260619 09:04:48.762269 19476 heartbeater.cc:503] Master 127.0.98.62:46403 was elected leader, sending a full tablet report...
I20260619 09:04:48.763463 19493 raft_consensus.cc:348] T babbe0a35ae94c7eb9ed547b1dfa4aba P 6aace6fb21ae40ce8d02a88f873ddbb6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f878dce091dc417fa027e5a300f44517" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34721 } } peers { permanent_uuid: "6aace6fb21ae40ce8d02a88f873ddbb6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44071 } } peers { permanent_uuid: "7379e7e0ad2b4ac08f40c3fe7e916f32" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34195 } }
I20260619 09:04:48.763584 19493 raft_consensus.cc:374] T babbe0a35ae94c7eb9ed547b1dfa4aba P 6aace6fb21ae40ce8d02a88f873ddbb6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 09:04:48.763613 19493 raft_consensus.cc:729] T babbe0a35ae94c7eb9ed547b1dfa4aba P 6aace6fb21ae40ce8d02a88f873ddbb6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6aace6fb21ae40ce8d02a88f873ddbb6, State: Initialized, Role: FOLLOWER
I20260619 09:04:48.763715 19493 consensus_queue.cc:260] T babbe0a35ae94c7eb9ed547b1dfa4aba P 6aace6fb21ae40ce8d02a88f873ddbb6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f878dce091dc417fa027e5a300f44517" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34721 } } peers { permanent_uuid: "6aace6fb21ae40ce8d02a88f873ddbb6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44071 } } peers { permanent_uuid: "7379e7e0ad2b4ac08f40c3fe7e916f32" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34195 } }
I20260619 09:04:48.763967 19214 heartbeater.cc:503] Master 127.0.98.62:46403 was elected leader, sending a full tablet report...
I20260619 09:04:48.763967 19493 ts_tablet_manager.cc:1428] T babbe0a35ae94c7eb9ed547b1dfa4aba P 6aace6fb21ae40ce8d02a88f873ddbb6: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
W20260619 09:04:48.826702 19346 tablet.cc:2369] T babbe0a35ae94c7eb9ed547b1dfa4aba P 7379e7e0ad2b4ac08f40c3fe7e916f32: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 09:04:48.870231 19498 raft_consensus.cc:482] T babbe0a35ae94c7eb9ed547b1dfa4aba P 6aace6fb21ae40ce8d02a88f873ddbb6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 09:04:48.870364 19498 raft_consensus.cc:504] T babbe0a35ae94c7eb9ed547b1dfa4aba P 6aace6fb21ae40ce8d02a88f873ddbb6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f878dce091dc417fa027e5a300f44517" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34721 } } peers { permanent_uuid: "6aace6fb21ae40ce8d02a88f873ddbb6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44071 } } peers { permanent_uuid: "7379e7e0ad2b4ac08f40c3fe7e916f32" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34195 } }
I20260619 09:04:48.870675 19498 leader_election.cc:288] T babbe0a35ae94c7eb9ed547b1dfa4aba P 6aace6fb21ae40ce8d02a88f873ddbb6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f878dce091dc417fa027e5a300f44517 (127.0.98.3:34721), 7379e7e0ad2b4ac08f40c3fe7e916f32 (127.0.98.2:34195)
I20260619 09:04:48.874449 19489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:04:48.870937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60203 (local address 127.0.98.3:34721)
0619 09:04:48.870989 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0619 09:04:48.870991 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:04:48.871002 (+ 11us) server_negotiation.cc:407] Connection header received
0619 09:04:48.871064 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:04:48.871065 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:04:48.871074 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:04:48.871137 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 09:04:48.871301 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:04:48.872057 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:04:48.873532 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:04:48.874096 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:04:48.874173 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:04:48.874183 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0619 09:04:48.874337 (+ 154us) server_negotiation.cc:299] Negotiation successful
0619 09:04:48.874376 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260619 09:04:48.874449 19488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:04:48.870930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39681 (local address 127.0.98.2:34195)
0619 09:04:48.870989 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0619 09:04:48.870992 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 09:04:48.871240 (+ 248us) server_negotiation.cc:407] Connection header received
0619 09:04:48.871272 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:04:48.871273 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:04:48.871286 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:04:48.871336 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 09:04:48.871445 (+ 109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:04:48.872142 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:04:48.873531 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:04:48.874101 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:04:48.874173 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:04:48.874183 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0619 09:04:48.874337 (+ 154us) server_negotiation.cc:299] Negotiation successful
0619 09:04:48.874392 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260619 09:04:48.874464 19216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:04:48.870880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34721 (local address 127.0.98.1:60203)
0619 09:04:48.870935 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0619 09:04:48.870947 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 09:04:48.870993 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:04:48.871152 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:04:48.871152 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:04:48.871154 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 09:04:48.871232 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:04:48.871234 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:04:48.872155 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:04:48.872156 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:04:48.873427 (+ 1271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:04:48.873431 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:04:48.874183 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:04:48.874184 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:04:48.874266 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:04:48.874271 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 09:04:48.874327 (+ 56us) client_negotiation.cc:240] Negotiation successful
0619 09:04:48.874369 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35,"mutex_wait_us":14}
I20260619 09:04:48.874471 19499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:04:48.870983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34195 (local address 127.0.98.1:39681)
0619 09:04:48.871165 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0619 09:04:48.871175 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 09:04:48.871230 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:04:48.871342 (+ 112us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:04:48.871342 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:04:48.871344 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 09:04:48.871390 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:04:48.871391 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:04:48.872213 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:04:48.872214 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:04:48.873427 (+ 1213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:04:48.873431 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:04:48.874183 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:04:48.874184 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:04:48.874266 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:04:48.874271 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 09:04:48.874327 (+ 56us) client_negotiation.cc:240] Negotiation successful
0619 09:04:48.874367 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"mutex_wait_us":28,"thread_start_us":108,"threads_started":1}
I20260619 09:04:48.874809 19430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "babbe0a35ae94c7eb9ed547b1dfa4aba" candidate_uuid: "6aace6fb21ae40ce8d02a88f873ddbb6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f878dce091dc417fa027e5a300f44517" is_pre_election: true
I20260619 09:04:48.874975 19430 raft_consensus.cc:2454] T babbe0a35ae94c7eb9ed547b1dfa4aba P f878dce091dc417fa027e5a300f44517 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6aace6fb21ae40ce8d02a88f873ddbb6 in term 0.
I20260619 09:04:48.874989 19299 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "babbe0a35ae94c7eb9ed547b1dfa4aba" candidate_uuid: "6aace6fb21ae40ce8d02a88f873ddbb6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7379e7e0ad2b4ac08f40c3fe7e916f32" is_pre_election: true
I20260619 09:04:48.875137 19299 raft_consensus.cc:2454] T babbe0a35ae94c7eb9ed547b1dfa4aba P 7379e7e0ad2b4ac08f40c3fe7e916f32 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6aace6fb21ae40ce8d02a88f873ddbb6 in term 0.
I20260619 09:04:48.875216 19101 leader_election.cc:302] T babbe0a35ae94c7eb9ed547b1dfa4aba P 6aace6fb21ae40ce8d02a88f873ddbb6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6aace6fb21ae40ce8d02a88f873ddbb6, f878dce091dc417fa027e5a300f44517; no voters:
I20260619 09:04:48.875341 19498 raft_consensus.cc:2781] T babbe0a35ae94c7eb9ed547b1dfa4aba P 6aace6fb21ae40ce8d02a88f873ddbb6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 09:04:48.875398 19498 raft_consensus.cc:482] T babbe0a35ae94c7eb9ed547b1dfa4aba P 6aace6fb21ae40ce8d02a88f873ddbb6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 09:04:48.875417 19498 raft_consensus.cc:3037] T babbe0a35ae94c7eb9ed547b1dfa4aba P 6aace6fb21ae40ce8d02a88f873ddbb6 [term 0 FOLLOWER]: Advancing to term 1
I20260619 09:04:48.876057 19498 raft_consensus.cc:504] T babbe0a35ae94c7eb9ed547b1dfa4aba P 6aace6fb21ae40ce8d02a88f873ddbb6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f878dce091dc417fa027e5a300f44517" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34721 } } peers { permanent_uuid: "6aace6fb21ae40ce8d02a88f873ddbb6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44071 } } peers { permanent_uuid: "7379e7e0ad2b4ac08f40c3fe7e916f32" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34195 } }
I20260619 09:04:48.876181 19498 leader_election.cc:288] T babbe0a35ae94c7eb9ed547b1dfa4aba P 6aace6fb21ae40ce8d02a88f873ddbb6 [CANDIDATE]: Term 1 election: Requested vote from peers f878dce091dc417fa027e5a300f44517 (127.0.98.3:34721), 7379e7e0ad2b4ac08f40c3fe7e916f32 (127.0.98.2:34195)
I20260619 09:04:48.876328 19430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "babbe0a35ae94c7eb9ed547b1dfa4aba" candidate_uuid: "6aace6fb21ae40ce8d02a88f873ddbb6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f878dce091dc417fa027e5a300f44517"
I20260619 09:04:48.876355 19299 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "babbe0a35ae94c7eb9ed547b1dfa4aba" candidate_uuid: "6aace6fb21ae40ce8d02a88f873ddbb6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7379e7e0ad2b4ac08f40c3fe7e916f32"
I20260619 09:04:48.876400 19430 raft_consensus.cc:3037] T babbe0a35ae94c7eb9ed547b1dfa4aba P f878dce091dc417fa027e5a300f44517 [term 0 FOLLOWER]: Advancing to term 1
I20260619 09:04:48.876423 19299 raft_consensus.cc:3037] T babbe0a35ae94c7eb9ed547b1dfa4aba P 7379e7e0ad2b4ac08f40c3fe7e916f32 [term 0 FOLLOWER]: Advancing to term 1
I20260619 09:04:48.877105 19430 raft_consensus.cc:2454] T babbe0a35ae94c7eb9ed547b1dfa4aba P f878dce091dc417fa027e5a300f44517 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6aace6fb21ae40ce8d02a88f873ddbb6 in term 1.
I20260619 09:04:48.877120 19299 raft_consensus.cc:2454] T babbe0a35ae94c7eb9ed547b1dfa4aba P 7379e7e0ad2b4ac08f40c3fe7e916f32 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6aace6fb21ae40ce8d02a88f873ddbb6 in term 1.
I20260619 09:04:48.877271 19101 leader_election.cc:302] T babbe0a35ae94c7eb9ed547b1dfa4aba P 6aace6fb21ae40ce8d02a88f873ddbb6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6aace6fb21ae40ce8d02a88f873ddbb6, f878dce091dc417fa027e5a300f44517; no voters:
I20260619 09:04:48.877355 19498 raft_consensus.cc:2781] T babbe0a35ae94c7eb9ed547b1dfa4aba P 6aace6fb21ae40ce8d02a88f873ddbb6 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 09:04:48.877485 19498 raft_consensus.cc:686] T babbe0a35ae94c7eb9ed547b1dfa4aba P 6aace6fb21ae40ce8d02a88f873ddbb6 [term 1 LEADER]: Becoming Leader. State: Replica: 6aace6fb21ae40ce8d02a88f873ddbb6, State: Running, Role: LEADER
I20260619 09:04:48.877571 19498 consensus_queue.cc:237] T babbe0a35ae94c7eb9ed547b1dfa4aba P 6aace6fb21ae40ce8d02a88f873ddbb6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f878dce091dc417fa027e5a300f44517" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34721 } } peers { permanent_uuid: "6aace6fb21ae40ce8d02a88f873ddbb6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44071 } } peers { permanent_uuid: "7379e7e0ad2b4ac08f40c3fe7e916f32" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34195 } }
I20260619 09:04:48.878330 19026 catalog_manager.cc:5495] T babbe0a35ae94c7eb9ed547b1dfa4aba P 6aace6fb21ae40ce8d02a88f873ddbb6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6aace6fb21ae40ce8d02a88f873ddbb6 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "6aace6fb21ae40ce8d02a88f873ddbb6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f878dce091dc417fa027e5a300f44517" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34721 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6aace6fb21ae40ce8d02a88f873ddbb6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44071 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7379e7e0ad2b4ac08f40c3fe7e916f32" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34195 } health_report { overall_health: UNKNOWN } } }
I20260619 09:04:48.897732 19487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:04:48.895579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46326 (local address 127.0.98.1:44071)
0619 09:04:48.895625 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0619 09:04:48.895627 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:04:48.895634 (+ 7us) server_negotiation.cc:407] Connection header received
0619 09:04:48.895665 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:04:48.895666 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:04:48.895683 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:04:48.895739 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 09:04:48.895858 (+ 119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:04:48.896363 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:04:48.897076 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:04:48.897268 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:04:48.897326 (+ 58us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:04:48.897440 (+ 114us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 09:04:48.897537 (+ 97us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 09:04:48.897606 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0619 09:04:48.897648 (+ 42us) server_negotiation.cc:299] Negotiation successful
0619 09:04:48.897655 (+ 7us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260619 09:04:48.899533 19299 raft_consensus.cc:1261] T babbe0a35ae94c7eb9ed547b1dfa4aba P 7379e7e0ad2b4ac08f40c3fe7e916f32 [term 1 FOLLOWER]: Refusing update from remote peer 6aace6fb21ae40ce8d02a88f873ddbb6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 09:04:48.899559 19430 raft_consensus.cc:1261] T babbe0a35ae94c7eb9ed547b1dfa4aba P f878dce091dc417fa027e5a300f44517 [term 1 FOLLOWER]: Refusing update from remote peer 6aace6fb21ae40ce8d02a88f873ddbb6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 09:04:48.899878 19500 consensus_queue.cc:1035] T babbe0a35ae94c7eb9ed547b1dfa4aba P 6aace6fb21ae40ce8d02a88f873ddbb6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7379e7e0ad2b4ac08f40c3fe7e916f32" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 09:04:48.900040 19498 consensus_queue.cc:1035] T babbe0a35ae94c7eb9ed547b1dfa4aba P 6aace6fb21ae40ce8d02a88f873ddbb6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f878dce091dc417fa027e5a300f44517" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 09:04:48.903173 19505 mvcc.cc:205] Tried to move back new op lower bound from 7298498104930603008 to 7298498104843825152. Current Snapshot: MvccSnapshot[applied={T|T < 7298498104930603008 or (T in {7298498104930603008})}]
I20260619 09:04:48.904672 19025 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34548:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 09:04:48.904753 19025 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34548:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 09:04:48.905457 19025 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5a180c40cf06444db06a290ede78e632: Sending DeleteTablet for 3 replicas of tablet babbe0a35ae94c7eb9ed547b1dfa4aba
I20260619 09:04:48.905833 19148 tablet_service.cc:1514] Processing DeleteTablet for tablet babbe0a35ae94c7eb9ed547b1dfa4aba with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 09:04:48 UTC) from {username='slave'} at 127.0.0.1:46314
I20260619 09:04:48.905833 19410 tablet_service.cc:1514] Processing DeleteTablet for tablet babbe0a35ae94c7eb9ed547b1dfa4aba with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 09:04:48 UTC) from {username='slave'} at 127.0.0.1:39528
I20260619 09:04:48.906025 19513 tablet_replica.cc:330] T babbe0a35ae94c7eb9ed547b1dfa4aba P f878dce091dc417fa027e5a300f44517: stopping tablet replica
I20260619 09:04:48.906018 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ez5NB/build/release/bin/kudu with pid 19086
I20260619 09:04:48.906051 19279 tablet_service.cc:1514] Processing DeleteTablet for tablet babbe0a35ae94c7eb9ed547b1dfa4aba with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 09:04:48 UTC) from {username='slave'} at 127.0.0.1:49968
I20260619 09:04:48.906119 19513 raft_consensus.cc:2229] T babbe0a35ae94c7eb9ed547b1dfa4aba P f878dce091dc417fa027e5a300f44517 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 09:04:48.906205 19513 raft_consensus.cc:2258] T babbe0a35ae94c7eb9ed547b1dfa4aba P f878dce091dc417fa027e5a300f44517 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 09:04:48.906997 19513 ts_tablet_manager.cc:1905] T babbe0a35ae94c7eb9ed547b1dfa4aba P f878dce091dc417fa027e5a300f44517: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 09:04:48.908491 19515 tablet_replica.cc:330] T babbe0a35ae94c7eb9ed547b1dfa4aba P 7379e7e0ad2b4ac08f40c3fe7e916f32: stopping tablet replica
I20260619 09:04:48.908597 19515 raft_consensus.cc:2229] T babbe0a35ae94c7eb9ed547b1dfa4aba P 7379e7e0ad2b4ac08f40c3fe7e916f32 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 09:04:48.908591 19513 ts_tablet_manager.cc:1918] T babbe0a35ae94c7eb9ed547b1dfa4aba P f878dce091dc417fa027e5a300f44517: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 09:04:48.908653 19513 log.cc:1196] T babbe0a35ae94c7eb9ed547b1dfa4aba P f878dce091dc417fa027e5a300f44517: Deleting WAL directory at /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781859658134510-392-0/minicluster-data/ts-2/wal/wals/babbe0a35ae94c7eb9ed547b1dfa4aba
I20260619 09:04:48.908697 19515 raft_consensus.cc:2258] T babbe0a35ae94c7eb9ed547b1dfa4aba P 7379e7e0ad2b4ac08f40c3fe7e916f32 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 09:04:48.908913 19513 ts_tablet_manager.cc:1939] T babbe0a35ae94c7eb9ed547b1dfa4aba P f878dce091dc417fa027e5a300f44517: Deleting consensus metadata
I20260619 09:04:48.909112 19515 ts_tablet_manager.cc:1905] T babbe0a35ae94c7eb9ed547b1dfa4aba P 7379e7e0ad2b4ac08f40c3fe7e916f32: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 09:04:48.909315 19010 catalog_manager.cc:4841] TS f878dce091dc417fa027e5a300f44517 (127.0.98.3:34721): tablet babbe0a35ae94c7eb9ed547b1dfa4aba (table test-table [id=fd521b8b8b5c4af5b7ac4bff8ad4b977]) successfully deleted
I20260619 09:04:48.910400 19515 ts_tablet_manager.cc:1918] T babbe0a35ae94c7eb9ed547b1dfa4aba P 7379e7e0ad2b4ac08f40c3fe7e916f32: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 09:04:48.910460 19515 log.cc:1196] T babbe0a35ae94c7eb9ed547b1dfa4aba P 7379e7e0ad2b4ac08f40c3fe7e916f32: Deleting WAL directory at /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781859658134510-392-0/minicluster-data/ts-1/wal/wals/babbe0a35ae94c7eb9ed547b1dfa4aba
I20260619 09:04:48.910655 19515 ts_tablet_manager.cc:1939] T babbe0a35ae94c7eb9ed547b1dfa4aba P 7379e7e0ad2b4ac08f40c3fe7e916f32: Deleting consensus metadata
I20260619 09:04:48.911087 19011 catalog_manager.cc:4841] TS 7379e7e0ad2b4ac08f40c3fe7e916f32 (127.0.98.2:34195): tablet babbe0a35ae94c7eb9ed547b1dfa4aba (table test-table [id=fd521b8b8b5c4af5b7ac4bff8ad4b977]) successfully deleted
W20260619 09:04:48.911839 19012 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:44071 (error 108)
I20260619 09:04:48.912043 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ez5NB/build/release/bin/kudu with pid 19217
I20260619 09:04:48.915212 19485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:04:48.915088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44071 (local address 127.0.0.1:46328)
0619 09:04:48.915141 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0619 09:04:48.915184 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260619 09:04:48.915336 19012 catalog_manager.cc:4584] TS 6aace6fb21ae40ce8d02a88f873ddbb6 (127.0.98.1:44071): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet babbe0a35ae94c7eb9ed547b1dfa4aba: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44071: connect: Connection refused (error 111)
I20260619 09:04:48.916572 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ez5NB/build/release/bin/kudu with pid 19348
I20260619 09:04:48.920853 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ez5NB/build/release/bin/kudu with pid 18995
2026-06-19T09:04:48Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (794 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781859658134510-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 09:04:48 dist-test-slave-mtdr krb5kdc[19519](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 09:04:48 dist-test-slave-mtdr krb5kdc[19519](info): set up 2 sockets
Jun 19 09:04:48 dist-test-slave-mtdr krb5kdc[19519](info): commencing operation
krb5kdc: starting...
W20260619 09:04:49.874444 392 mini_kdc.cc:121] Time spent starting KDC: real 0.934s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781859658134510-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781859658134510-392-0/krb5kdc/test-user.keytab.
Jun 19 09:04:49 dist-test-slave-mtdr krb5kdc[19519](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859889, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T09:04:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T09:04:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781859658134510-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781859658134510-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 09:04:49.985621 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ez5NB/build/release/bin/kudu
/tmp/dist-test-task4ez5NB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781859658134510-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781859658134510-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781859658134510-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781859658134510-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34433
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781859658134510-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46371
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34433
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781859658134510-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781859658134510-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781859658134510-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 09:04:50.066365 19532 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 09:04:50.066560 19532 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 09:04:50.066589 19532 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 09:04:50.068048 19532 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 09:04:50.068109 19532 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 09:04:50.068128 19532 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 09:04:50.068146 19532 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 09:04:50.068163 19532 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 09:04:50.070279 19532 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46371
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781859658134510-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781859658134510-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34433
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781859658134510-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34433
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781859658134510-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19532
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781859658134510-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 09:04:50.070533 19532 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 09:04:50.070744 19532 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 09:04:50.075241 19540 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:04:50.075237 19538 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:04:50.075351 19532 server_base.cc:1034] running on GCE node
W20260619 09:04:50.075452 19537 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:04:50.075666 19532 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 09:04:50.075961 19532 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 09:04:50.077098 19532 hybrid_clock.cc:648] HybridClock initialized: now 1781859890077083 us; error 32 us; skew 500 ppm
Jun 19 09:04:50 dist-test-slave-mtdr krb5kdc[19519](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859890, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 09:04:50.079324 19532 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 09:04:50.079726 19532 webserver.cc:460] Webserver started at http://127.0.98.62:40635/ using document root <none> and password file <none>
I20260619 09:04:50.079914 19532 fs_manager.cc:359] Metadata directory not provided
I20260619 09:04:50.079952 19532 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 09:04:50.080036 19532 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 09:04:50.080982 19532 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781859658134510-392-0/minicluster-data/master-0/data/instance:
uuid: "5f29f455b5b54865b8158e549ea67e2a"
format_stamp: "Formatted at 2026-06-19 09:04:50 on dist-test-slave-mtdr"
server_key: "337b56494dfa7a7bd31e1a8307b638c0"
server_key_iv: "6eaf4bfe7004ea6642decc319b8523ad"
server_key_version: "encryptionkey@0"
I20260619 09:04:50.081259 19532 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781859658134510-392-0/minicluster-data/master-0/wal/instance:
uuid: "5f29f455b5b54865b8158e549ea67e2a"
format_stamp: "Formatted at 2026-06-19 09:04:50 on dist-test-slave-mtdr"
server_key: "337b56494dfa7a7bd31e1a8307b638c0"
server_key_iv: "6eaf4bfe7004ea6642decc319b8523ad"
server_key_version: "encryptionkey@0"
I20260619 09:04:50.082407 19532 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 09:04:50.083096 19547 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 09:04:50.083251 19532 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 09:04:50.083334 19532 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781859658134510-392-0/minicluster-data/master-0/data,/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781859658134510-392-0/minicluster-data/master-0/wal
uuid: "5f29f455b5b54865b8158e549ea67e2a"
format_stamp: "Formatted at 2026-06-19 09:04:50 on dist-test-slave-mtdr"
server_key: "337b56494dfa7a7bd31e1a8307b638c0"
server_key_iv: "6eaf4bfe7004ea6642decc319b8523ad"
server_key_version: "encryptionkey@0"
I20260619 09:04:50.083385 19532 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781859658134510-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781859658134510-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781859658134510-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 09:04:50.103739 19532 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 09:04:50.105651 19532 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 09:04:50.105759 19532 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 09:04:50.109489 19532 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34433
I20260619 09:04:50.109557 19599 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34433 every 8 connection(s)
I20260619 09:04:50.109859 19532 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781859658134510-392-0/minicluster-data/master-0/data/info.pb
I20260619 09:04:50.110994 19600 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 09:04:50.113461 19600 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f29f455b5b54865b8158e549ea67e2a: Bootstrap starting.
I20260619 09:04:50.113998 19600 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5f29f455b5b54865b8158e549ea67e2a: Neither blocks nor log segments found. Creating new log.
I20260619 09:04:50.114241 19600 log.cc:824] T 00000000000000000000000000000000 P 5f29f455b5b54865b8158e549ea67e2a: Log is configured to *not* fsync() on all Append() calls
I20260619 09:04:50.114949 19600 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f29f455b5b54865b8158e549ea67e2a: No bootstrap required, opened a new log
I20260619 09:04:50.116461 19600 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5f29f455b5b54865b8158e549ea67e2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f29f455b5b54865b8158e549ea67e2a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34433 } }
I20260619 09:04:50.116592 19600 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5f29f455b5b54865b8158e549ea67e2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 09:04:50.116622 19600 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5f29f455b5b54865b8158e549ea67e2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f29f455b5b54865b8158e549ea67e2a, State: Initialized, Role: FOLLOWER
I20260619 09:04:50.116753 19600 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5f29f455b5b54865b8158e549ea67e2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f29f455b5b54865b8158e549ea67e2a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34433 } }
I20260619 09:04:50.116812 19600 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5f29f455b5b54865b8158e549ea67e2a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 09:04:50.116839 19600 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5f29f455b5b54865b8158e549ea67e2a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 09:04:50.116884 19600 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5f29f455b5b54865b8158e549ea67e2a [term 0 FOLLOWER]: Advancing to term 1
I20260619 09:04:50.117350 19600 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5f29f455b5b54865b8158e549ea67e2a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f29f455b5b54865b8158e549ea67e2a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34433 } }
I20260619 09:04:50.117442 19600 leader_election.cc:302] T 00000000000000000000000000000000 P 5f29f455b5b54865b8158e549ea67e2a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5f29f455b5b54865b8158e549ea67e2a; no voters:
I20260619 09:04:50.117594 19600 leader_election.cc:288] T 00000000000000000000000000000000 P 5f29f455b5b54865b8158e549ea67e2a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 09:04:50.117650 19603 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5f29f455b5b54865b8158e549ea67e2a [term 1 FOLLOWER]: Leader election won for term 1
I20260619 09:04:50.117802 19603 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5f29f455b5b54865b8158e549ea67e2a [term 1 LEADER]: Becoming Leader. State: Replica: 5f29f455b5b54865b8158e549ea67e2a, State: Running, Role: LEADER
I20260619 09:04:50.117834 19600 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5f29f455b5b54865b8158e549ea67e2a [sys.catalog]: configured and running, proceeding with master startup.
I20260619 09:04:50.117884 19603 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5f29f455b5b54865b8158e549ea67e2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f29f455b5b54865b8158e549ea67e2a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34433 } }
I20260619 09:04:50.118237 19605 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f29f455b5b54865b8158e549ea67e2a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5f29f455b5b54865b8158e549ea67e2a. Latest consensus state: current_term: 1 leader_uuid: "5f29f455b5b54865b8158e549ea67e2a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f29f455b5b54865b8158e549ea67e2a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34433 } } }
I20260619 09:04:50.118309 19605 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f29f455b5b54865b8158e549ea67e2a [sys.catalog]: This master's current role is: LEADER
I20260619 09:04:50.118357 19604 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f29f455b5b54865b8158e549ea67e2a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5f29f455b5b54865b8158e549ea67e2a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f29f455b5b54865b8158e549ea67e2a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34433 } } }
I20260619 09:04:50.118404 19604 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f29f455b5b54865b8158e549ea67e2a [sys.catalog]: This master's current role is: LEADER
I20260619 09:04:50.118842 19611 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 09:04:50.119292 19611 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 09:04:50.119850 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ez5NB/build/release/bin/kudu as pid 19532
I20260619 09:04:50.119961 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781859658134510-392-0/minicluster-data/master-0/wal/instance
I20260619 09:04:50.120170 392 external_mini_cluster.cc:1428] Setting key 19517c6367d05051f93430a92d9c12ea
I20260619 09:04:50.121170 19611 catalog_manager.cc:1347] Generated new cluster ID: 6928e6d853ca4532b21c816187f0efd6
I20260619 09:04:50.121212 19611 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 09:04:50.126400 19611 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 09:04:50.126957 19611 catalog_manager.cc:1504] Loading token signing keys...
Jun 19 09:04:50 dist-test-slave-mtdr krb5kdc[19519](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859889, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 09:04:50.131513 19611 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5f29f455b5b54865b8158e549ea67e2a: Generated new TSK 0
I20260619 09:04:50.131672 19611 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 09:04:50.133086 19622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:04:50.120586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59440 (local address 127.0.98.62:34433)
0619 09:04:50.120875 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0619 09:04:50.120883 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 09:04:50.120908 (+ 25us) server_negotiation.cc:407] Connection header received
0619 09:04:50.120996 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:04:50.121001 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:04:50.121041 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:04:50.122955 (+ 1914us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:04:50.123704 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:04:50.125140 (+ 1436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:04:50.126236 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:04:50.126893 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:04:50.127002 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:04:50.129730 (+ 2728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:04:50.129734 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:04:50.129734 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:04:50.129746 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:04:50.131585 (+ 1839us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:04:50.131946 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:04:50.131950 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:04:50.131950 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:04:50.132062 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:04:50.132235 (+ 173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:04:50.132235 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:04:50.132236 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:04:50.132513 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:04:50.132628 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0619 09:04:50.132707 (+ 79us) server_negotiation.cc:299] Negotiation successful
0619 09:04:50.132858 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":250,"thread_start_us":66,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781859658134510-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781859658134510-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 09:04:50.157447 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ez5NB/build/release/bin/kudu
/tmp/dist-test-task4ez5NB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781859658134510-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781859658134510-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781859658134510-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781859658134510-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34433
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781859658134510-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46371
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781859658134510-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781859658134510-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781859658134510-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 09:04:50.235411 19626 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 09:04:50.235607 19626 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 09:04:50.235630 19626 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 09:04:50.236759 19626 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 09:04:50.236799 19626 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 09:04:50.236824 19626 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 09:04:50.238271 19626 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46371
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781859658134510-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781859658134510-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781859658134510-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781859658134510-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34433
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19626
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781859658134510-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 09:04:50.238473 19626 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 09:04:50.238687 19626 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 09:04:50.243427 19632 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:04:50.243410 19631 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:04:50.243427 19634 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:04:50.243567 19626 server_base.cc:1034] running on GCE node
I20260619 09:04:50.243824 19626 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 09:04:50.244058 19626 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 09:04:50.245222 19626 hybrid_clock.cc:648] HybridClock initialized: now 1781859890245208 us; error 28 us; skew 500 ppm
Jun 19 09:04:50 dist-test-slave-mtdr krb5kdc[19519](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859890, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 09:04:50.247673 19626 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 09:04:50.248059 19626 webserver.cc:460] Webserver started at http://127.0.98.1:46651/ using document root <none> and password file <none>
I20260619 09:04:50.248235 19626 fs_manager.cc:359] Metadata directory not provided
I20260619 09:04:50.248275 19626 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 09:04:50.248358 19626 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 09:04:50.249328 19626 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781859658134510-392-0/minicluster-data/ts-0/data/instance:
uuid: "db5d8a4119734d91accdeaa231664281"
format_stamp: "Formatted at 2026-06-19 09:04:50 on dist-test-slave-mtdr"
server_key: "240f74d09a986a51105aa2cb32cdbede"
server_key_iv: "140f81761ca0fc0b0eb1ccc941f57360"
server_key_version: "encryptionkey@0"
I20260619 09:04:50.249604 19626 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781859658134510-392-0/minicluster-data/ts-0/wal/instance:
uuid: "db5d8a4119734d91accdeaa231664281"
format_stamp: "Formatted at 2026-06-19 09:04:50 on dist-test-slave-mtdr"
server_key: "240f74d09a986a51105aa2cb32cdbede"
server_key_iv: "140f81761ca0fc0b0eb1ccc941f57360"
server_key_version: "encryptionkey@0"
I20260619 09:04:50.250738 19626 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260619 09:04:50.251458 19641 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 09:04:50.251629 19626 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 09:04:50.251704 19626 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781859658134510-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781859658134510-392-0/minicluster-data/ts-0/wal
uuid: "db5d8a4119734d91accdeaa231664281"
format_stamp: "Formatted at 2026-06-19 09:04:50 on dist-test-slave-mtdr"
server_key: "240f74d09a986a51105aa2cb32cdbede"
server_key_iv: "140f81761ca0fc0b0eb1ccc941f57360"
server_key_version: "encryptionkey@0"
I20260619 09:04:50.251770 19626 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781859658134510-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781859658134510-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781859658134510-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 09:04:50.260535 19626 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 09:04:50.262691 19626 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 09:04:50.262859 19626 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 09:04:50.263088 19626 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 09:04:50.263396 19626 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 09:04:50.263432 19626 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 09:04:50.263474 19626 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 09:04:50.263502 19626 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 09:04:50.269404 19626 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:33725
I20260619 09:04:50.269439 19754 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:33725 every 8 connection(s)
I20260619 09:04:50.269752 19626 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781859658134510-392-0/minicluster-data/ts-0/data/info.pb
I20260619 09:04:50.271718 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ez5NB/build/release/bin/kudu as pid 19626
I20260619 09:04:50.271831 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781859658134510-392-0/minicluster-data/ts-0/wal/instance
I20260619 09:04:50.272046 392 external_mini_cluster.cc:1428] Setting key 0e255efab0b2407b3a7088e118e794f4
Jun 19 09:04:50 dist-test-slave-mtdr krb5kdc[19519](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859890, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 09:04:50.279239 19622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:04:50.270411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59935 (local address 127.0.98.62:34433)
0619 09:04:50.270502 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0619 09:04:50.270504 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:04:50.270652 (+ 148us) server_negotiation.cc:407] Connection header received
0619 09:04:50.271608 (+ 956us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:04:50.271610 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:04:50.271623 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:04:50.271700 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:04:50.272396 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:04:50.273068 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:04:50.274192 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:04:50.274381 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:04:50.274430 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:04:50.276627 (+ 2197us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:04:50.276628 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:04:50.276629 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:04:50.276636 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:04:50.277914 (+ 1278us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:04:50.278316 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:04:50.278318 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:04:50.278318 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:04:50.278376 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:04:50.278599 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:04:50.278600 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:04:50.278600 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:04:50.278707 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:04:50.278762 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0619 09:04:50.279063 (+ 301us) server_negotiation.cc:299] Negotiation successful
0619 09:04:50.279149 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 09:04:50.279379 19757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:04:50.270375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34433 (local address 127.0.98.1:59935)
0619 09:04:50.270551 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0619 09:04:50.270587 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0619 09:04:50.271482 (+ 895us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:04:50.271756 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:04:50.271759 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:04:50.272062 (+ 303us) client_negotiation.cc:189] Negotiated authn=SASL
0619 09:04:50.272311 (+ 249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:04:50.272318 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:04:50.273152 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:04:50.273153 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:04:50.274109 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:04:50.274112 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:04:50.274447 (+ 335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:04:50.274447 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:04:50.274582 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:04:50.275091 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 09:04:50.275100 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0619 09:04:50.276541 (+ 1441us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 09:04:50.277996 (+ 1455us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:04:50.278009 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:04:50.278011 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:04:50.278249 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:04:50.278428 (+ 179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:04:50.278430 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:04:50.278430 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:04:50.278540 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:04:50.278762 (+ 222us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 09:04:50.278768 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 09:04:50.278939 (+ 171us) client_negotiation.cc:769] Sending connection context
0619 09:04:50.279032 (+ 93us) client_negotiation.cc:240] Negotiation successful
0619 09:04:50.279175 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":75,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260619 09:04:50.279891 19755 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34433
I20260619 09:04:50.279991 19755 heartbeater.cc:464] Registering TS with master...
I20260619 09:04:50.280150 19755 heartbeater.cc:511] Master 127.0.98.62:34433 requested a full tablet report, sending...
I20260619 09:04:50.280619 19564 ts_manager.cc:194] Registered new tserver with Master: db5d8a4119734d91accdeaa231664281 (127.0.98.1:33725)
I20260619 09:04:50.281188 19564 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:59935
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781859658134510-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781859658134510-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 09:04:50.297133 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ez5NB/build/release/bin/kudu
/tmp/dist-test-task4ez5NB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781859658134510-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781859658134510-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781859658134510-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781859658134510-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34433
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781859658134510-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46371
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781859658134510-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781859658134510-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781859658134510-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 09:04:50.374356 19760 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 09:04:50.374558 19760 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 09:04:50.374579 19760 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 09:04:50.375718 19760 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 09:04:50.375761 19760 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 09:04:50.375785 19760 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 09:04:50.377200 19760 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46371
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781859658134510-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781859658134510-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781859658134510-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781859658134510-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34433
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19760
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781859658134510-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 09:04:50.377406 19760 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 09:04:50.377622 19760 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 09:04:50.382301 19766 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:04:50.382301 19768 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:04:50.382308 19765 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:04:50.382543 19760 server_base.cc:1034] running on GCE node
I20260619 09:04:50.382756 19760 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 09:04:50.383000 19760 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 09:04:50.384150 19760 hybrid_clock.cc:648] HybridClock initialized: now 1781859890384137 us; error 31 us; skew 500 ppm
Jun 19 09:04:50 dist-test-slave-mtdr krb5kdc[19519](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859890, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 09:04:50.386682 19760 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 09:04:50.387176 19760 webserver.cc:460] Webserver started at http://127.0.98.2:39685/ using document root <none> and password file <none>
I20260619 09:04:50.387369 19760 fs_manager.cc:359] Metadata directory not provided
I20260619 09:04:50.387415 19760 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 09:04:50.387539 19760 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 09:04:50.388535 19760 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781859658134510-392-0/minicluster-data/ts-1/data/instance:
uuid: "33adc50975044646abf749ba79b73920"
format_stamp: "Formatted at 2026-06-19 09:04:50 on dist-test-slave-mtdr"
server_key: "9036a7ff082de46bee1965a1a031e1eb"
server_key_iv: "7f93b431af2d5b9fce1c90839bc65fe9"
server_key_version: "encryptionkey@0"
I20260619 09:04:50.388849 19760 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781859658134510-392-0/minicluster-data/ts-1/wal/instance:
uuid: "33adc50975044646abf749ba79b73920"
format_stamp: "Formatted at 2026-06-19 09:04:50 on dist-test-slave-mtdr"
server_key: "9036a7ff082de46bee1965a1a031e1eb"
server_key_iv: "7f93b431af2d5b9fce1c90839bc65fe9"
server_key_version: "encryptionkey@0"
I20260619 09:04:50.390084 19760 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 09:04:50.390776 19775 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 09:04:50.390938 19760 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 09:04:50.391026 19760 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781859658134510-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781859658134510-392-0/minicluster-data/ts-1/wal
uuid: "33adc50975044646abf749ba79b73920"
format_stamp: "Formatted at 2026-06-19 09:04:50 on dist-test-slave-mtdr"
server_key: "9036a7ff082de46bee1965a1a031e1eb"
server_key_iv: "7f93b431af2d5b9fce1c90839bc65fe9"
server_key_version: "encryptionkey@0"
I20260619 09:04:50.391086 19760 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781859658134510-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781859658134510-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781859658134510-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 09:04:50.406427 19760 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 09:04:50.408551 19760 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 09:04:50.408685 19760 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 09:04:50.408900 19760 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 09:04:50.409188 19760 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 09:04:50.409221 19760 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 09:04:50.409250 19760 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 09:04:50.409273 19760 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 09:04:50.414762 19760 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41211
I20260619 09:04:50.414839 19888 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41211 every 8 connection(s)
I20260619 09:04:50.415120 19760 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781859658134510-392-0/minicluster-data/ts-1/data/info.pb
Jun 19 09:04:50 dist-test-slave-mtdr krb5kdc[19519](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859890, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 09:04:50.421504 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ez5NB/build/release/bin/kudu as pid 19760
I20260619 09:04:50.421627 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781859658134510-392-0/minicluster-data/ts-1/wal/instance
I20260619 09:04:50.421850 392 external_mini_cluster.cc:1428] Setting key ba1c8dd52207ce41c4334f8b8a1bcbc1
I20260619 09:04:50.424229 19622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:04:50.415767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55307 (local address 127.0.98.62:34433)
0619 09:04:50.415837 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0619 09:04:50.415839 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:04:50.416017 (+ 178us) server_negotiation.cc:407] Connection header received
0619 09:04:50.416907 (+ 890us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:04:50.416908 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:04:50.416923 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:04:50.416987 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:04:50.417570 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:04:50.418064 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:04:50.419076 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:04:50.419257 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:04:50.419308 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:04:50.421340 (+ 2032us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:04:50.421342 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:04:50.421342 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:04:50.421350 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:04:50.422985 (+ 1635us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:04:50.423336 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:04:50.423338 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:04:50.423338 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:04:50.423410 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:04:50.423606 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:04:50.423606 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:04:50.423607 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:04:50.423733 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:04:50.423806 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0619 09:04:50.424063 (+ 257us) server_negotiation.cc:299] Negotiation successful
0619 09:04:50.424142 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260619 09:04:50.424388 19891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:04:50.415753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34433 (local address 127.0.98.2:55307)
0619 09:04:50.415905 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0619 09:04:50.415945 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 09:04:50.416806 (+ 861us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:04:50.417044 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:04:50.417047 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:04:50.417303 (+ 256us) client_negotiation.cc:189] Negotiated authn=SASL
0619 09:04:50.417494 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:04:50.417499 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:04:50.418149 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:04:50.418150 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:04:50.419016 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:04:50.419019 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:04:50.419319 (+ 300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:04:50.419320 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:04:50.419435 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:04:50.419873 (+ 438us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 09:04:50.419881 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 09:04:50.421214 (+ 1333us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 09:04:50.423057 (+ 1843us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:04:50.423063 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:04:50.423064 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:04:50.423269 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:04:50.423473 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:04:50.423474 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:04:50.423474 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:04:50.423557 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:04:50.423793 (+ 236us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 09:04:50.423795 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 09:04:50.423913 (+ 118us) client_negotiation.cc:769] Sending connection context
0619 09:04:50.424018 (+ 105us) client_negotiation.cc:240] Negotiation successful
0619 09:04:50.424164 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":81,"threads_started":1}
I20260619 09:04:50.424988 19889 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34433
I20260619 09:04:50.425067 19889 heartbeater.cc:464] Registering TS with master...
I20260619 09:04:50.425235 19889 heartbeater.cc:511] Master 127.0.98.62:34433 requested a full tablet report, sending...
I20260619 09:04:50.425583 19564 ts_manager.cc:194] Registered new tserver with Master: 33adc50975044646abf749ba79b73920 (127.0.98.2:41211)
I20260619 09:04:50.425938 19564 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:55307
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781859658134510-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781859658134510-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 09:04:50.445703 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ez5NB/build/release/bin/kudu
/tmp/dist-test-task4ez5NB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781859658134510-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781859658134510-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781859658134510-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781859658134510-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34433
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781859658134510-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46371
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781859658134510-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781859658134510-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781859658134510-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 09:04:50.523857 19894 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 09:04:50.524070 19894 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 09:04:50.524102 19894 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 09:04:50.525245 19894 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 09:04:50.525285 19894 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 09:04:50.525319 19894 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 09:04:50.526777 19894 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46371
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781859658134510-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781859658134510-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781859658134510-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781859658134510-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34433
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19894
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781859658134510-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 09:04:50.527021 19894 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 09:04:50.527258 19894 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 09:04:50.531977 19899 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:04:50.531977 19902 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:04:50.531971 19900 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:04:50.532191 19894 server_base.cc:1034] running on GCE node
I20260619 09:04:50.532413 19894 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 09:04:50.532646 19894 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 09:04:50.533787 19894 hybrid_clock.cc:648] HybridClock initialized: now 1781859890533770 us; error 33 us; skew 500 ppm
Jun 19 09:04:50 dist-test-slave-mtdr krb5kdc[19519](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859890, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 09:04:50.536213 19894 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 09:04:50.536635 19894 webserver.cc:460] Webserver started at http://127.0.98.3:46561/ using document root <none> and password file <none>
I20260619 09:04:50.536839 19894 fs_manager.cc:359] Metadata directory not provided
I20260619 09:04:50.536885 19894 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 09:04:50.536985 19894 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 09:04:50.537985 19894 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781859658134510-392-0/minicluster-data/ts-2/data/instance:
uuid: "8556fa2e599f49c9ba73a004bb0f39c5"
format_stamp: "Formatted at 2026-06-19 09:04:50 on dist-test-slave-mtdr"
server_key: "3e9692add734f0de4312b9ca7522d9a1"
server_key_iv: "6f2fb040b19afbea2795bcd0fac2c9be"
server_key_version: "encryptionkey@0"
I20260619 09:04:50.538338 19894 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781859658134510-392-0/minicluster-data/ts-2/wal/instance:
uuid: "8556fa2e599f49c9ba73a004bb0f39c5"
format_stamp: "Formatted at 2026-06-19 09:04:50 on dist-test-slave-mtdr"
server_key: "3e9692add734f0de4312b9ca7522d9a1"
server_key_iv: "6f2fb040b19afbea2795bcd0fac2c9be"
server_key_version: "encryptionkey@0"
I20260619 09:04:50.539608 19894 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 09:04:50.540270 19909 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 09:04:50.540395 19894 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 09:04:50.540467 19894 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781859658134510-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781859658134510-392-0/minicluster-data/ts-2/wal
uuid: "8556fa2e599f49c9ba73a004bb0f39c5"
format_stamp: "Formatted at 2026-06-19 09:04:50 on dist-test-slave-mtdr"
server_key: "3e9692add734f0de4312b9ca7522d9a1"
server_key_iv: "6f2fb040b19afbea2795bcd0fac2c9be"
server_key_version: "encryptionkey@0"
I20260619 09:04:50.540525 19894 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781859658134510-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781859658134510-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781859658134510-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 09:04:50.556695 19894 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 09:04:50.558748 19894 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 09:04:50.558898 19894 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 09:04:50.559100 19894 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 09:04:50.559369 19894 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 09:04:50.559402 19894 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 09:04:50.559429 19894 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 09:04:50.559453 19894 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 09:04:50.564651 19894 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42087
I20260619 09:04:50.564709 20022 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42087 every 8 connection(s)
I20260619 09:04:50.564980 19894 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781859658134510-392-0/minicluster-data/ts-2/data/info.pb
I20260619 09:04:50.570214 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ez5NB/build/release/bin/kudu as pid 19894
I20260619 09:04:50.570325 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781859658134510-392-0/minicluster-data/ts-2/wal/instance
I20260619 09:04:50.570508 392 external_mini_cluster.cc:1428] Setting key 14bcb887fd1edaf4693893e05f08f38b
Jun 19 09:04:50 dist-test-slave-mtdr krb5kdc[19519](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859890, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 09:04:50.573447 19622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:04:50.565676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58641 (local address 127.0.98.62:34433)
0619 09:04:50.565737 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0619 09:04:50.565739 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:04:50.565911 (+ 172us) server_negotiation.cc:407] Connection header received
0619 09:04:50.566804 (+ 893us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:04:50.566805 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:04:50.566847 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:04:50.566903 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:04:50.567517 (+ 614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:04:50.568020 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:04:50.568957 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:04:50.569160 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:04:50.569213 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:04:50.571328 (+ 2115us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:04:50.571329 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:04:50.571329 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:04:50.571337 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:04:50.572381 (+ 1044us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:04:50.572721 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:04:50.572722 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:04:50.572722 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:04:50.572774 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:04:50.572954 (+ 180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:04:50.572954 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:04:50.572955 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:04:50.573048 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:04:50.573092 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0619 09:04:50.573319 (+ 227us) server_negotiation.cc:299] Negotiation successful
0619 09:04:50.573385 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260619 09:04:50.573545 20025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:04:50.565637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34433 (local address 127.0.98.3:58641)
0619 09:04:50.565793 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0619 09:04:50.565835 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 09:04:50.566704 (+ 869us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:04:50.566980 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:04:50.566983 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:04:50.567249 (+ 266us) client_negotiation.cc:189] Negotiated authn=SASL
0619 09:04:50.567430 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:04:50.567435 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:04:50.568100 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:04:50.568101 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:04:50.568882 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:04:50.568884 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:04:50.569223 (+ 339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:04:50.569224 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:04:50.569333 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:04:50.569704 (+ 371us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 09:04:50.569712 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 09:04:50.571241 (+ 1529us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 09:04:50.572453 (+ 1212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:04:50.572458 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:04:50.572459 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:04:50.572661 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:04:50.572823 (+ 162us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:04:50.572824 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:04:50.572824 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:04:50.572912 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:04:50.573088 (+ 176us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 09:04:50.573090 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 09:04:50.573194 (+ 104us) client_negotiation.cc:769] Sending connection context
0619 09:04:50.573266 (+ 72us) client_negotiation.cc:240] Negotiation successful
0619 09:04:50.573376 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":75,"threads_started":1}
I20260619 09:04:50.573896 20023 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34433
I20260619 09:04:50.573973 20023 heartbeater.cc:464] Registering TS with master...
I20260619 09:04:50.574126 20023 heartbeater.cc:511] Master 127.0.98.62:34433 requested a full tablet report, sending...
I20260619 09:04:50.574455 19564 ts_manager.cc:194] Registered new tserver with Master: 8556fa2e599f49c9ba73a004bb0f39c5 (127.0.98.3:42087)
I20260619 09:04:50.574800 19564 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:58641
I20260619 09:04:50.581945 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 09:04:50.588532 19622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:04:50.583203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59444 (local address 127.0.98.62:34433)
0619 09:04:50.583258 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0619 09:04:50.583260 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:04:50.583336 (+ 76us) server_negotiation.cc:407] Connection header received
0619 09:04:50.583415 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:04:50.583416 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:04:50.583424 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:04:50.583465 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:04:50.583943 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:04:50.584401 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:04:50.585129 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:04:50.585293 (+ 164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:04:50.585331 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:04:50.586446 (+ 1115us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:04:50.586447 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:04:50.586448 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:04:50.586453 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:04:50.587687 (+ 1234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:04:50.587965 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:04:50.587966 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:04:50.587966 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:04:50.588008 (+ 42us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:04:50.588153 (+ 145us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:04:50.588154 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:04:50.588154 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:04:50.588266 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:04:50.588329 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0619 09:04:50.588422 (+ 93us) server_negotiation.cc:299] Negotiation successful
0619 09:04:50.588472 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260619 09:04:50.589877 19564 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59444:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 09:04:50.591487 19564 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 09:04:50.597594 20031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:04:50.593473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41211 (local address 127.0.0.1:36522)
0619 09:04:50.593845 (+ 372us) negotiation.cc:107] Waiting for socket to connect
0619 09:04:50.593852 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0619 09:04:50.593883 (+ 31us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:04:50.594182 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:04:50.594183 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:04:50.594186 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 09:04:50.594239 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:04:50.594240 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:04:50.595587 (+ 1347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:04:50.595588 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:04:50.596498 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:04:50.596500 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:04:50.597306 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:04:50.597307 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:04:50.597380 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:04:50.597396 (+ 16us) client_negotiation.cc:769] Sending connection context
0619 09:04:50.597458 (+ 62us) client_negotiation.cc:240] Negotiation successful
0619 09:04:50.597511 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":86,"threads_started":1}
I20260619 09:04:50.597702 20032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:04:50.593452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42087 (local address 127.0.0.1:57714)
0619 09:04:50.593669 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0619 09:04:50.593710 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 09:04:50.593770 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:04:50.594051 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:04:50.594053 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:04:50.594055 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 09:04:50.594116 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:04:50.594118 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:04:50.595424 (+ 1306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:04:50.595424 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:04:50.596763 (+ 1339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:04:50.596765 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:04:50.597431 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:04:50.597432 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:04:50.597495 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:04:50.597499 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 09:04:50.597561 (+ 62us) client_negotiation.cc:240] Negotiation successful
0619 09:04:50.597602 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":199,"mutex_wait_us":59,"thread_start_us":62,"threads_started":1}
I20260619 09:04:50.597846 20036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:04:50.593879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36522 (local address 127.0.98.2:41211)
0619 09:04:50.594019 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0619 09:04:50.594031 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 09:04:50.594040 (+ 9us) server_negotiation.cc:407] Connection header received
0619 09:04:50.594073 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:04:50.594076 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:04:50.594090 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:04:50.594169 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 09:04:50.594406 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:04:50.595499 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:04:50.596565 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:04:50.597222 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:04:50.597293 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:04:50.597352 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0619 09:04:50.597721 (+ 369us) server_negotiation.cc:299] Negotiation successful
0619 09:04:50.597770 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105,"thread_start_us":74,"threads_started":1}
I20260619 09:04:50.598026 20033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:04:50.593451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33725 (local address 127.0.0.1:42574)
0619 09:04:50.593669 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0619 09:04:50.593710 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 09:04:50.593770 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:04:50.594402 (+ 632us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:04:50.594403 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:04:50.594405 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 09:04:50.594444 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:04:50.594446 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:04:50.595388 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:04:50.595389 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:04:50.596733 (+ 1344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:04:50.596735 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:04:50.597800 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:04:50.597801 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:04:50.597871 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:04:50.597875 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 09:04:50.597929 (+ 54us) client_negotiation.cc:240] Negotiation successful
0619 09:04:50.597967 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":181,"mutex_wait_us":88,"thread_start_us":60,"threads_started":1}
I20260619 09:04:50.598170 20035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:04:50.593716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57714 (local address 127.0.98.3:42087)
0619 09:04:50.593881 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0619 09:04:50.593886 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 09:04:50.593896 (+ 10us) server_negotiation.cc:407] Connection header received
0619 09:04:50.593935 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:04:50.593937 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:04:50.593953 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:04:50.594040 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 09:04:50.594201 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:04:50.595342 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:04:50.596846 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:04:50.597359 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:04:50.597429 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:04:50.597485 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0619 09:04:50.597578 (+ 93us) server_negotiation.cc:299] Negotiation successful
0619 09:04:50.597632 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":65,"threads_started":1}
I20260619 09:04:50.598172 20034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:04:50.593696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42574 (local address 127.0.98.1:33725)
0619 09:04:50.594186 (+ 490us) server_negotiation.cc:206] Beginning negotiation
0619 09:04:50.594190 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 09:04:50.594199 (+ 9us) server_negotiation.cc:407] Connection header received
0619 09:04:50.594236 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:04:50.594238 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:04:50.594252 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:04:50.594320 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 09:04:50.594518 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:04:50.595296 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:04:50.596827 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:04:50.597520 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:04:50.597588 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:04:50.597639 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0619 09:04:50.598047 (+ 408us) server_negotiation.cc:299] Negotiation successful
0619 09:04:50.598090 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":449,"thread_start_us":72,"threads_started":1}
I20260619 09:04:50.598510 19823 tablet_service.cc:1467] Processing CreateTablet for tablet 1713f8b488684f04b9b16ab2c7ae78fe (DEFAULT_TABLE table=test-table [id=d3b1f842bba84d248f9cb2ea8227e984]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 09:04:50.598687 19957 tablet_service.cc:1467] Processing CreateTablet for tablet 1713f8b488684f04b9b16ab2c7ae78fe (DEFAULT_TABLE table=test-table [id=d3b1f842bba84d248f9cb2ea8227e984]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 09:04:50.598897 19823 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1713f8b488684f04b9b16ab2c7ae78fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 09:04:50.598778 19689 tablet_service.cc:1467] Processing CreateTablet for tablet 1713f8b488684f04b9b16ab2c7ae78fe (DEFAULT_TABLE table=test-table [id=d3b1f842bba84d248f9cb2ea8227e984]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 09:04:50.598978 19957 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1713f8b488684f04b9b16ab2c7ae78fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 09:04:50.599126 19689 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1713f8b488684f04b9b16ab2c7ae78fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 09:04:50.601603 20037 tablet_bootstrap.cc:492] T 1713f8b488684f04b9b16ab2c7ae78fe P db5d8a4119734d91accdeaa231664281: Bootstrap starting.
I20260619 09:04:50.602175 20039 tablet_bootstrap.cc:492] T 1713f8b488684f04b9b16ab2c7ae78fe P 8556fa2e599f49c9ba73a004bb0f39c5: Bootstrap starting.
I20260619 09:04:50.602214 20037 tablet_bootstrap.cc:654] T 1713f8b488684f04b9b16ab2c7ae78fe P db5d8a4119734d91accdeaa231664281: Neither blocks nor log segments found. Creating new log.
I20260619 09:04:50.602726 20037 log.cc:824] T 1713f8b488684f04b9b16ab2c7ae78fe P db5d8a4119734d91accdeaa231664281: Log is configured to *not* fsync() on all Append() calls
I20260619 09:04:50.602833 20039 tablet_bootstrap.cc:654] T 1713f8b488684f04b9b16ab2c7ae78fe P 8556fa2e599f49c9ba73a004bb0f39c5: Neither blocks nor log segments found. Creating new log.
I20260619 09:04:50.602864 20038 tablet_bootstrap.cc:492] T 1713f8b488684f04b9b16ab2c7ae78fe P 33adc50975044646abf749ba79b73920: Bootstrap starting.
I20260619 09:04:50.603353 20039 log.cc:824] T 1713f8b488684f04b9b16ab2c7ae78fe P 8556fa2e599f49c9ba73a004bb0f39c5: Log is configured to *not* fsync() on all Append() calls
I20260619 09:04:50.603488 20038 tablet_bootstrap.cc:654] T 1713f8b488684f04b9b16ab2c7ae78fe P 33adc50975044646abf749ba79b73920: Neither blocks nor log segments found. Creating new log.
I20260619 09:04:50.603543 20037 tablet_bootstrap.cc:492] T 1713f8b488684f04b9b16ab2c7ae78fe P db5d8a4119734d91accdeaa231664281: No bootstrap required, opened a new log
I20260619 09:04:50.603613 20037 ts_tablet_manager.cc:1397] T 1713f8b488684f04b9b16ab2c7ae78fe P db5d8a4119734d91accdeaa231664281: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260619 09:04:50.603910 20039 tablet_bootstrap.cc:492] T 1713f8b488684f04b9b16ab2c7ae78fe P 8556fa2e599f49c9ba73a004bb0f39c5: No bootstrap required, opened a new log
I20260619 09:04:50.603951 20039 ts_tablet_manager.cc:1397] T 1713f8b488684f04b9b16ab2c7ae78fe P 8556fa2e599f49c9ba73a004bb0f39c5: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 09:04:50.604004 20038 log.cc:824] T 1713f8b488684f04b9b16ab2c7ae78fe P 33adc50975044646abf749ba79b73920: Log is configured to *not* fsync() on all Append() calls
I20260619 09:04:50.604744 20038 tablet_bootstrap.cc:492] T 1713f8b488684f04b9b16ab2c7ae78fe P 33adc50975044646abf749ba79b73920: No bootstrap required, opened a new log
I20260619 09:04:50.604799 20038 ts_tablet_manager.cc:1397] T 1713f8b488684f04b9b16ab2c7ae78fe P 33adc50975044646abf749ba79b73920: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260619 09:04:50.605448 20039 raft_consensus.cc:348] T 1713f8b488684f04b9b16ab2c7ae78fe P 8556fa2e599f49c9ba73a004bb0f39c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db5d8a4119734d91accdeaa231664281" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33725 } } peers { permanent_uuid: "8556fa2e599f49c9ba73a004bb0f39c5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42087 } } peers { permanent_uuid: "33adc50975044646abf749ba79b73920" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41211 } }
I20260619 09:04:50.605581 20039 raft_consensus.cc:374] T 1713f8b488684f04b9b16ab2c7ae78fe P 8556fa2e599f49c9ba73a004bb0f39c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 09:04:50.605602 20039 raft_consensus.cc:729] T 1713f8b488684f04b9b16ab2c7ae78fe P 8556fa2e599f49c9ba73a004bb0f39c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8556fa2e599f49c9ba73a004bb0f39c5, State: Initialized, Role: FOLLOWER
I20260619 09:04:50.605504 20037 raft_consensus.cc:348] T 1713f8b488684f04b9b16ab2c7ae78fe P db5d8a4119734d91accdeaa231664281 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db5d8a4119734d91accdeaa231664281" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33725 } } peers { permanent_uuid: "8556fa2e599f49c9ba73a004bb0f39c5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42087 } } peers { permanent_uuid: "33adc50975044646abf749ba79b73920" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41211 } }
I20260619 09:04:50.605674 20037 raft_consensus.cc:374] T 1713f8b488684f04b9b16ab2c7ae78fe P db5d8a4119734d91accdeaa231664281 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 09:04:50.605707 20037 raft_consensus.cc:729] T 1713f8b488684f04b9b16ab2c7ae78fe P db5d8a4119734d91accdeaa231664281 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db5d8a4119734d91accdeaa231664281, State: Initialized, Role: FOLLOWER
I20260619 09:04:50.605692 20039 consensus_queue.cc:260] T 1713f8b488684f04b9b16ab2c7ae78fe P 8556fa2e599f49c9ba73a004bb0f39c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db5d8a4119734d91accdeaa231664281" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33725 } } peers { permanent_uuid: "8556fa2e599f49c9ba73a004bb0f39c5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42087 } } peers { permanent_uuid: "33adc50975044646abf749ba79b73920" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41211 } }
I20260619 09:04:50.605816 20037 consensus_queue.cc:260] T 1713f8b488684f04b9b16ab2c7ae78fe P db5d8a4119734d91accdeaa231664281 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db5d8a4119734d91accdeaa231664281" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33725 } } peers { permanent_uuid: "8556fa2e599f49c9ba73a004bb0f39c5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42087 } } peers { permanent_uuid: "33adc50975044646abf749ba79b73920" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41211 } }
I20260619 09:04:50.605916 20039 ts_tablet_manager.cc:1428] T 1713f8b488684f04b9b16ab2c7ae78fe P 8556fa2e599f49c9ba73a004bb0f39c5: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 09:04:50.606006 20023 heartbeater.cc:503] Master 127.0.98.62:34433 was elected leader, sending a full tablet report...
I20260619 09:04:50.606070 20037 ts_tablet_manager.cc:1428] T 1713f8b488684f04b9b16ab2c7ae78fe P db5d8a4119734d91accdeaa231664281: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260619 09:04:50.606099 19755 heartbeater.cc:503] Master 127.0.98.62:34433 was elected leader, sending a full tablet report...
I20260619 09:04:50.607240 20038 raft_consensus.cc:348] T 1713f8b488684f04b9b16ab2c7ae78fe P 33adc50975044646abf749ba79b73920 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db5d8a4119734d91accdeaa231664281" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33725 } } peers { permanent_uuid: "8556fa2e599f49c9ba73a004bb0f39c5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42087 } } peers { permanent_uuid: "33adc50975044646abf749ba79b73920" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41211 } }
I20260619 09:04:50.607366 20038 raft_consensus.cc:374] T 1713f8b488684f04b9b16ab2c7ae78fe P 33adc50975044646abf749ba79b73920 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 09:04:50.607396 20038 raft_consensus.cc:729] T 1713f8b488684f04b9b16ab2c7ae78fe P 33adc50975044646abf749ba79b73920 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33adc50975044646abf749ba79b73920, State: Initialized, Role: FOLLOWER
I20260619 09:04:50.607498 20038 consensus_queue.cc:260] T 1713f8b488684f04b9b16ab2c7ae78fe P 33adc50975044646abf749ba79b73920 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db5d8a4119734d91accdeaa231664281" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33725 } } peers { permanent_uuid: "8556fa2e599f49c9ba73a004bb0f39c5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42087 } } peers { permanent_uuid: "33adc50975044646abf749ba79b73920" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41211 } }
I20260619 09:04:50.607710 20038 ts_tablet_manager.cc:1428] T 1713f8b488684f04b9b16ab2c7ae78fe P 33adc50975044646abf749ba79b73920: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260619 09:04:50.607765 19889 heartbeater.cc:503] Master 127.0.98.62:34433 was elected leader, sending a full tablet report...
W20260619 09:04:50.665640 19890 tablet.cc:2369] T 1713f8b488684f04b9b16ab2c7ae78fe P 33adc50975044646abf749ba79b73920: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 09:04:50.718227 20043 raft_consensus.cc:482] T 1713f8b488684f04b9b16ab2c7ae78fe P 8556fa2e599f49c9ba73a004bb0f39c5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 09:04:50.718379 20043 raft_consensus.cc:504] T 1713f8b488684f04b9b16ab2c7ae78fe P 8556fa2e599f49c9ba73a004bb0f39c5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db5d8a4119734d91accdeaa231664281" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33725 } } peers { permanent_uuid: "8556fa2e599f49c9ba73a004bb0f39c5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42087 } } peers { permanent_uuid: "33adc50975044646abf749ba79b73920" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41211 } }
I20260619 09:04:50.718693 20043 leader_election.cc:288] T 1713f8b488684f04b9b16ab2c7ae78fe P 8556fa2e599f49c9ba73a004bb0f39c5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db5d8a4119734d91accdeaa231664281 (127.0.98.1:33725), 33adc50975044646abf749ba79b73920 (127.0.98.2:41211)
I20260619 09:04:50.721379 20034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:04:50.718969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38337 (local address 127.0.98.1:33725)
0619 09:04:50.719008 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0619 09:04:50.719010 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:04:50.719079 (+ 69us) server_negotiation.cc:407] Connection header received
0619 09:04:50.719135 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:04:50.719137 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:04:50.719150 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:04:50.719198 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 09:04:50.719333 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:04:50.719834 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:04:50.720722 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:04:50.721074 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:04:50.721131 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:04:50.721141 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0619 09:04:50.721274 (+ 133us) server_negotiation.cc:299] Negotiation successful
0619 09:04:50.721311 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260619 09:04:50.721379 20025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:04:50.718900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33725 (local address 127.0.98.3:38337)
0619 09:04:50.718993 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0619 09:04:50.719004 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 09:04:50.719074 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:04:50.719210 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:04:50.719211 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:04:50.719212 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 09:04:50.719274 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:04:50.719275 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:04:50.719912 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:04:50.719913 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:04:50.720635 (+ 722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:04:50.720637 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:04:50.721145 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:04:50.721146 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:04:50.721211 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:04:50.721214 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 09:04:50.721259 (+ 45us) client_negotiation.cc:240] Negotiation successful
0619 09:04:50.721300 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":75,"mutex_wait_us":1}
I20260619 09:04:50.721470 20036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:04:50.718949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57907 (local address 127.0.98.2:41211)
0619 09:04:50.718999 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0619 09:04:50.719001 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:04:50.719203 (+ 202us) server_negotiation.cc:407] Connection header received
0619 09:04:50.719252 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:04:50.719253 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:04:50.719265 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:04:50.719312 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 09:04:50.719458 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:04:50.719903 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:04:50.720796 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:04:50.721157 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:04:50.721217 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:04:50.721227 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0619 09:04:50.721373 (+ 146us) server_negotiation.cc:299] Negotiation successful
0619 09:04:50.721408 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260619 09:04:50.721644 20046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:04:50.718912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41211 (local address 127.0.98.3:57907)
0619 09:04:50.719123 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0619 09:04:50.719132 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 09:04:50.719197 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:04:50.719325 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:04:50.719326 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:04:50.719328 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 09:04:50.719398 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:04:50.719399 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:04:50.719981 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:04:50.719982 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:04:50.720709 (+ 727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:04:50.720710 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:04:50.721236 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:04:50.721237 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:04:50.721309 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:04:50.721312 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 09:04:50.721358 (+ 46us) client_negotiation.cc:240] Negotiation successful
0619 09:04:50.721407 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":123,"threads_started":1}
I20260619 09:04:50.721685 19709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1713f8b488684f04b9b16ab2c7ae78fe" candidate_uuid: "8556fa2e599f49c9ba73a004bb0f39c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db5d8a4119734d91accdeaa231664281" is_pre_election: true
I20260619 09:04:50.721850 19709 raft_consensus.cc:2454] T 1713f8b488684f04b9b16ab2c7ae78fe P db5d8a4119734d91accdeaa231664281 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8556fa2e599f49c9ba73a004bb0f39c5 in term 0.
I20260619 09:04:50.722002 19843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1713f8b488684f04b9b16ab2c7ae78fe" candidate_uuid: "8556fa2e599f49c9ba73a004bb0f39c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33adc50975044646abf749ba79b73920" is_pre_election: true
I20260619 09:04:50.722097 19911 leader_election.cc:302] T 1713f8b488684f04b9b16ab2c7ae78fe P 8556fa2e599f49c9ba73a004bb0f39c5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8556fa2e599f49c9ba73a004bb0f39c5, db5d8a4119734d91accdeaa231664281; no voters:
I20260619 09:04:50.722139 19843 raft_consensus.cc:2454] T 1713f8b488684f04b9b16ab2c7ae78fe P 33adc50975044646abf749ba79b73920 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8556fa2e599f49c9ba73a004bb0f39c5 in term 0.
I20260619 09:04:50.722220 20043 raft_consensus.cc:2781] T 1713f8b488684f04b9b16ab2c7ae78fe P 8556fa2e599f49c9ba73a004bb0f39c5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 09:04:50.722296 20043 raft_consensus.cc:482] T 1713f8b488684f04b9b16ab2c7ae78fe P 8556fa2e599f49c9ba73a004bb0f39c5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 09:04:50.722322 20043 raft_consensus.cc:3037] T 1713f8b488684f04b9b16ab2c7ae78fe P 8556fa2e599f49c9ba73a004bb0f39c5 [term 0 FOLLOWER]: Advancing to term 1
I20260619 09:04:50.723037 20043 raft_consensus.cc:504] T 1713f8b488684f04b9b16ab2c7ae78fe P 8556fa2e599f49c9ba73a004bb0f39c5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db5d8a4119734d91accdeaa231664281" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33725 } } peers { permanent_uuid: "8556fa2e599f49c9ba73a004bb0f39c5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42087 } } peers { permanent_uuid: "33adc50975044646abf749ba79b73920" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41211 } }
I20260619 09:04:50.723150 20043 leader_election.cc:288] T 1713f8b488684f04b9b16ab2c7ae78fe P 8556fa2e599f49c9ba73a004bb0f39c5 [CANDIDATE]: Term 1 election: Requested vote from peers db5d8a4119734d91accdeaa231664281 (127.0.98.1:33725), 33adc50975044646abf749ba79b73920 (127.0.98.2:41211)
I20260619 09:04:50.723342 19709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1713f8b488684f04b9b16ab2c7ae78fe" candidate_uuid: "8556fa2e599f49c9ba73a004bb0f39c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db5d8a4119734d91accdeaa231664281"
I20260619 09:04:50.723379 19843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1713f8b488684f04b9b16ab2c7ae78fe" candidate_uuid: "8556fa2e599f49c9ba73a004bb0f39c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33adc50975044646abf749ba79b73920"
I20260619 09:04:50.723416 19709 raft_consensus.cc:3037] T 1713f8b488684f04b9b16ab2c7ae78fe P db5d8a4119734d91accdeaa231664281 [term 0 FOLLOWER]: Advancing to term 1
I20260619 09:04:50.723444 19843 raft_consensus.cc:3037] T 1713f8b488684f04b9b16ab2c7ae78fe P 33adc50975044646abf749ba79b73920 [term 0 FOLLOWER]: Advancing to term 1
I20260619 09:04:50.724084 19843 raft_consensus.cc:2454] T 1713f8b488684f04b9b16ab2c7ae78fe P 33adc50975044646abf749ba79b73920 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8556fa2e599f49c9ba73a004bb0f39c5 in term 1.
I20260619 09:04:50.724138 19709 raft_consensus.cc:2454] T 1713f8b488684f04b9b16ab2c7ae78fe P db5d8a4119734d91accdeaa231664281 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8556fa2e599f49c9ba73a004bb0f39c5 in term 1.
I20260619 09:04:50.724273 19910 leader_election.cc:302] T 1713f8b488684f04b9b16ab2c7ae78fe P 8556fa2e599f49c9ba73a004bb0f39c5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 33adc50975044646abf749ba79b73920, 8556fa2e599f49c9ba73a004bb0f39c5; no voters:
I20260619 09:04:50.724392 20043 raft_consensus.cc:2781] T 1713f8b488684f04b9b16ab2c7ae78fe P 8556fa2e599f49c9ba73a004bb0f39c5 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 09:04:50.724526 20043 raft_consensus.cc:686] T 1713f8b488684f04b9b16ab2c7ae78fe P 8556fa2e599f49c9ba73a004bb0f39c5 [term 1 LEADER]: Becoming Leader. State: Replica: 8556fa2e599f49c9ba73a004bb0f39c5, State: Running, Role: LEADER
I20260619 09:04:50.724617 20043 consensus_queue.cc:237] T 1713f8b488684f04b9b16ab2c7ae78fe P 8556fa2e599f49c9ba73a004bb0f39c5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db5d8a4119734d91accdeaa231664281" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33725 } } peers { permanent_uuid: "8556fa2e599f49c9ba73a004bb0f39c5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42087 } } peers { permanent_uuid: "33adc50975044646abf749ba79b73920" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41211 } }
I20260619 09:04:50.725394 19564 catalog_manager.cc:5495] T 1713f8b488684f04b9b16ab2c7ae78fe P 8556fa2e599f49c9ba73a004bb0f39c5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8556fa2e599f49c9ba73a004bb0f39c5 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "8556fa2e599f49c9ba73a004bb0f39c5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db5d8a4119734d91accdeaa231664281" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33725 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8556fa2e599f49c9ba73a004bb0f39c5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42087 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "33adc50975044646abf749ba79b73920" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41211 } health_report { overall_health: UNKNOWN } } }
I20260619 09:04:50.742565 20035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:04:50.740386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57716 (local address 127.0.98.3:42087)
0619 09:04:50.740433 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0619 09:04:50.740434 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 09:04:50.740446 (+ 12us) server_negotiation.cc:407] Connection header received
0619 09:04:50.740512 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:04:50.740513 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:04:50.740528 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:04:50.740588 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 09:04:50.740685 (+ 97us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:04:50.741131 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:04:50.741858 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:04:50.742043 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:04:50.742086 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:04:50.742197 (+ 111us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 09:04:50.742292 (+ 95us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 09:04:50.742347 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0619 09:04:50.742443 (+ 96us) server_negotiation.cc:299] Negotiation successful
0619 09:04:50.742458 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260619 09:04:50.744365 19843 raft_consensus.cc:1261] T 1713f8b488684f04b9b16ab2c7ae78fe P 33adc50975044646abf749ba79b73920 [term 1 FOLLOWER]: Refusing update from remote peer 8556fa2e599f49c9ba73a004bb0f39c5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 09:04:50.744416 19709 raft_consensus.cc:1261] T 1713f8b488684f04b9b16ab2c7ae78fe P db5d8a4119734d91accdeaa231664281 [term 1 FOLLOWER]: Refusing update from remote peer 8556fa2e599f49c9ba73a004bb0f39c5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 09:04:50.744722 20047 consensus_queue.cc:1035] T 1713f8b488684f04b9b16ab2c7ae78fe P 8556fa2e599f49c9ba73a004bb0f39c5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "33adc50975044646abf749ba79b73920" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 09:04:50.744845 20043 consensus_queue.cc:1035] T 1713f8b488684f04b9b16ab2c7ae78fe P 8556fa2e599f49c9ba73a004bb0f39c5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db5d8a4119734d91accdeaa231664281" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 09:04:50.747249 20051 mvcc.cc:205] Tried to move back new op lower bound from 7298498112486576128 to 7298498112408952832. Current Snapshot: MvccSnapshot[applied={T|T < 7298498112486576128}]
I20260619 09:04:50.749095 19564 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59444:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 09:04:50.749185 19564 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59444:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 09:04:50.749358 20053 mvcc.cc:205] Tried to move back new op lower bound from 7298498112486576128 to 7298498112408952832. Current Snapshot: MvccSnapshot[applied={T|T < 7298498112486576128}]
I20260619 09:04:50.749863 19564 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5f29f455b5b54865b8158e549ea67e2a: Sending DeleteTablet for 3 replicas of tablet 1713f8b488684f04b9b16ab2c7ae78fe
I20260619 09:04:50.750175 19957 tablet_service.cc:1514] Processing DeleteTablet for tablet 1713f8b488684f04b9b16ab2c7ae78fe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 09:04:50 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57714
I20260619 09:04:50.750248 19689 tablet_service.cc:1514] Processing DeleteTablet for tablet 1713f8b488684f04b9b16ab2c7ae78fe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 09:04:50 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42574
I20260619 09:04:50.750332 19823 tablet_service.cc:1514] Processing DeleteTablet for tablet 1713f8b488684f04b9b16ab2c7ae78fe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 09:04:50 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36522
I20260619 09:04:50.750434 20061 tablet_replica.cc:330] T 1713f8b488684f04b9b16ab2c7ae78fe P db5d8a4119734d91accdeaa231664281: stopping tablet replica
I20260619 09:04:50.750526 20061 raft_consensus.cc:2229] T 1713f8b488684f04b9b16ab2c7ae78fe P db5d8a4119734d91accdeaa231664281 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 09:04:50.750550 20060 tablet_replica.cc:330] T 1713f8b488684f04b9b16ab2c7ae78fe P 8556fa2e599f49c9ba73a004bb0f39c5: stopping tablet replica
I20260619 09:04:50.750635 20061 raft_consensus.cc:2258] T 1713f8b488684f04b9b16ab2c7ae78fe P db5d8a4119734d91accdeaa231664281 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 09:04:50.750712 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ez5NB/build/release/bin/kudu with pid 19626
I20260619 09:04:50.752584 20060 raft_consensus.cc:2229] T 1713f8b488684f04b9b16ab2c7ae78fe P 8556fa2e599f49c9ba73a004bb0f39c5 [term 1 LEADER]: Raft consensus shutting down.
I20260619 09:04:50.752694 20060 raft_consensus.cc:2258] T 1713f8b488684f04b9b16ab2c7ae78fe P 8556fa2e599f49c9ba73a004bb0f39c5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 09:04:50.753129 20060 ts_tablet_manager.cc:1905] T 1713f8b488684f04b9b16ab2c7ae78fe P 8556fa2e599f49c9ba73a004bb0f39c5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 09:04:50.754791 20060 ts_tablet_manager.cc:1918] T 1713f8b488684f04b9b16ab2c7ae78fe P 8556fa2e599f49c9ba73a004bb0f39c5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 09:04:50.754895 20060 log.cc:1196] T 1713f8b488684f04b9b16ab2c7ae78fe P 8556fa2e599f49c9ba73a004bb0f39c5: Deleting WAL directory at /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781859658134510-392-0/minicluster-data/ts-2/wal/wals/1713f8b488684f04b9b16ab2c7ae78fe
I20260619 09:04:50.755172 20060 ts_tablet_manager.cc:1939] T 1713f8b488684f04b9b16ab2c7ae78fe P 8556fa2e599f49c9ba73a004bb0f39c5: Deleting consensus metadata
I20260619 09:04:50.755575 19550 catalog_manager.cc:4841] TS 8556fa2e599f49c9ba73a004bb0f39c5 (127.0.98.3:42087): tablet 1713f8b488684f04b9b16ab2c7ae78fe (table test-table [id=d3b1f842bba84d248f9cb2ea8227e984]) successfully deleted
W20260619 09:04:50.756099 19549 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:33725 (error 108)
I20260619 09:04:50.756299 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ez5NB/build/release/bin/kudu with pid 19760
I20260619 09:04:50.756320 20062 tablet_replica.cc:330] T 1713f8b488684f04b9b16ab2c7ae78fe P 33adc50975044646abf749ba79b73920: stopping tablet replica
I20260619 09:04:50.761848 20032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:04:50.761687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41211 (local address 127.0.0.1:36530)
0619 09:04:50.761760 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0619 09:04:50.761798 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:41211: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260619 09:04:50.761852 20033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:04:50.761691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33725 (local address 127.0.0.1:42584)
0619 09:04:50.761776 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0619 09:04:50.761800 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71,"mutex_wait_us":22}
I20260619 09:04:50.761919 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ez5NB/build/release/bin/kudu with pid 19894
W20260619 09:04:50.762003 19548 catalog_manager.cc:4584] TS 33adc50975044646abf749ba79b73920 (127.0.98.2:41211): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1713f8b488684f04b9b16ab2c7ae78fe: Network error: Client connection negotiation failed: client connection to 127.0.98.2:41211: connect: Connection refused (error 111)
I20260619 09:04:50.766124 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ez5NB/build/release/bin/kudu with pid 19532
2026-06-19T09:04:50Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (1843 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-06-19T09:04:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T09:04:50Z Disabled control of system clock
I20260619 09:04:50.790324 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ez5NB/build/release/bin/kudu
/tmp/dist-test-task4ez5NB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781859658134510-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781859658134510-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781859658134510-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781859658134510-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38129
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:40953
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38129
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 09:04:50.868920 20067 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 09:04:50.869138 20067 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 09:04:50.869158 20067 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 09:04:50.870226 20067 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 09:04:50.870260 20067 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 09:04:50.870270 20067 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 09:04:50.870281 20067 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 09:04:50.870291 20067 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 09:04:50.871574 20067 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40953
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781859658134510-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781859658134510-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38129
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38129
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781859658134510-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20067
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781859658134510-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 09:04:50.871760 20067 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 09:04:50.871989 20067 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 09:04:50.876458 20072 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:04:50.876571 20067 server_base.cc:1034] running on GCE node
W20260619 09:04:50.876459 20073 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:04:50.876497 20075 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:04:50.876935 20067 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 09:04:50.877169 20067 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 09:04:50.878319 20067 hybrid_clock.cc:648] HybridClock initialized: now 1781859890878302 us; error 26 us; skew 500 ppm
I20260619 09:04:50.879480 20067 webserver.cc:460] Webserver started at http://127.0.98.62:43207/ using document root <none> and password file <none>
I20260619 09:04:50.879699 20067 fs_manager.cc:359] Metadata directory not provided
I20260619 09:04:50.879745 20067 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 09:04:50.879871 20067 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 09:04:50.880883 20067 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781859658134510-392-0/minicluster-data/master-0/data/instance:
uuid: "131add0a010c4580a9162628fd6d45f0"
format_stamp: "Formatted at 2026-06-19 09:04:50 on dist-test-slave-mtdr"
server_key: "19e9129aa85f3644fe4575108bc8fadc"
server_key_iv: "eacd02a207a2d2a9c060521dc9eb2b97"
server_key_version: "encryptionkey@0"
I20260619 09:04:50.881186 20067 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781859658134510-392-0/minicluster-data/master-0/wal/instance:
uuid: "131add0a010c4580a9162628fd6d45f0"
format_stamp: "Formatted at 2026-06-19 09:04:50 on dist-test-slave-mtdr"
server_key: "19e9129aa85f3644fe4575108bc8fadc"
server_key_iv: "eacd02a207a2d2a9c060521dc9eb2b97"
server_key_version: "encryptionkey@0"
I20260619 09:04:50.882329 20067 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 09:04:50.883062 20081 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 09:04:50.883219 20067 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 09:04:50.883296 20067 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781859658134510-392-0/minicluster-data/master-0/data,/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781859658134510-392-0/minicluster-data/master-0/wal
uuid: "131add0a010c4580a9162628fd6d45f0"
format_stamp: "Formatted at 2026-06-19 09:04:50 on dist-test-slave-mtdr"
server_key: "19e9129aa85f3644fe4575108bc8fadc"
server_key_iv: "eacd02a207a2d2a9c060521dc9eb2b97"
server_key_version: "encryptionkey@0"
I20260619 09:04:50.883356 20067 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781859658134510-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781859658134510-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781859658134510-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 09:04:50.904263 20067 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 09:04:50.904560 20067 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 09:04:50.904690 20067 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 09:04:50.908447 20067 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38129
I20260619 09:04:50.908499 20133 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38129 every 8 connection(s)
I20260619 09:04:50.908831 20067 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781859658134510-392-0/minicluster-data/master-0/data/info.pb
I20260619 09:04:50.909776 20134 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 09:04:50.912060 20134 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 131add0a010c4580a9162628fd6d45f0: Bootstrap starting.
I20260619 09:04:50.912596 20134 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 131add0a010c4580a9162628fd6d45f0: Neither blocks nor log segments found. Creating new log.
I20260619 09:04:50.912865 20134 log.cc:824] T 00000000000000000000000000000000 P 131add0a010c4580a9162628fd6d45f0: Log is configured to *not* fsync() on all Append() calls
I20260619 09:04:50.913533 20134 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 131add0a010c4580a9162628fd6d45f0: No bootstrap required, opened a new log
I20260619 09:04:50.914324 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ez5NB/build/release/bin/kudu as pid 20067
I20260619 09:04:50.914446 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781859658134510-392-0/minicluster-data/master-0/wal/instance
I20260619 09:04:50.914678 392 external_mini_cluster.cc:1428] Setting key 33c338b082751c6ed46f5f3aa1e2d0f6
I20260619 09:04:50.915685 20134 raft_consensus.cc:348] T 00000000000000000000000000000000 P 131add0a010c4580a9162628fd6d45f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "131add0a010c4580a9162628fd6d45f0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38129 } }
I20260619 09:04:50.915856 20134 raft_consensus.cc:374] T 00000000000000000000000000000000 P 131add0a010c4580a9162628fd6d45f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 09:04:50.915894 20134 raft_consensus.cc:729] T 00000000000000000000000000000000 P 131add0a010c4580a9162628fd6d45f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 131add0a010c4580a9162628fd6d45f0, State: Initialized, Role: FOLLOWER
I20260619 09:04:50.916030 20134 consensus_queue.cc:260] T 00000000000000000000000000000000 P 131add0a010c4580a9162628fd6d45f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "131add0a010c4580a9162628fd6d45f0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38129 } }
I20260619 09:04:50.916118 20134 raft_consensus.cc:388] T 00000000000000000000000000000000 P 131add0a010c4580a9162628fd6d45f0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 09:04:50.916153 20134 raft_consensus.cc:482] T 00000000000000000000000000000000 P 131add0a010c4580a9162628fd6d45f0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 09:04:50.916208 20134 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 131add0a010c4580a9162628fd6d45f0 [term 0 FOLLOWER]: Advancing to term 1
I20260619 09:04:50.916740 20134 raft_consensus.cc:504] T 00000000000000000000000000000000 P 131add0a010c4580a9162628fd6d45f0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "131add0a010c4580a9162628fd6d45f0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38129 } }
I20260619 09:04:50.916841 20134 leader_election.cc:302] T 00000000000000000000000000000000 P 131add0a010c4580a9162628fd6d45f0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 131add0a010c4580a9162628fd6d45f0; no voters:
I20260619 09:04:50.917006 20134 leader_election.cc:288] T 00000000000000000000000000000000 P 131add0a010c4580a9162628fd6d45f0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 09:04:50.917050 20139 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 131add0a010c4580a9162628fd6d45f0 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 09:04:50.917187 20139 raft_consensus.cc:686] T 00000000000000000000000000000000 P 131add0a010c4580a9162628fd6d45f0 [term 1 LEADER]: Becoming Leader. State: Replica: 131add0a010c4580a9162628fd6d45f0, State: Running, Role: LEADER
I20260619 09:04:50.917254 20134 sys_catalog.cc:564] T 00000000000000000000000000000000 P 131add0a010c4580a9162628fd6d45f0 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 09:04:50.917321 20139 consensus_queue.cc:237] T 00000000000000000000000000000000 P 131add0a010c4580a9162628fd6d45f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "131add0a010c4580a9162628fd6d45f0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38129 } }
I20260619 09:04:50.917644 20139 sys_catalog.cc:455] T 00000000000000000000000000000000 P 131add0a010c4580a9162628fd6d45f0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 131add0a010c4580a9162628fd6d45f0. Latest consensus state: current_term: 1 leader_uuid: "131add0a010c4580a9162628fd6d45f0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "131add0a010c4580a9162628fd6d45f0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38129 } } }
I20260619 09:04:50.917737 20139 sys_catalog.cc:458] T 00000000000000000000000000000000 P 131add0a010c4580a9162628fd6d45f0 [sys.catalog]: This master's current role is: LEADER
I20260619 09:04:50.917702 20140 sys_catalog.cc:455] T 00000000000000000000000000000000 P 131add0a010c4580a9162628fd6d45f0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "131add0a010c4580a9162628fd6d45f0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "131add0a010c4580a9162628fd6d45f0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38129 } } }
I20260619 09:04:50.917850 20140 sys_catalog.cc:458] T 00000000000000000000000000000000 P 131add0a010c4580a9162628fd6d45f0 [sys.catalog]: This master's current role is: LEADER
W20260619 09:04:50.919133 20155 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 131add0a010c4580a9162628fd6d45f0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 09:04:50.919247 20155 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 09:04:50.919348 20146 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 09:04:50.919552 20146 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 09:04:50.920753 20138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:04:50.915215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53358 (local address 127.0.98.62:38129)
0619 09:04:50.915403 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0619 09:04:50.915410 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 09:04:50.915434 (+ 24us) server_negotiation.cc:407] Connection header received
0619 09:04:50.915653 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:04:50.915658 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:04:50.915712 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:04:50.915819 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:04:50.916100 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:04:50.917629 (+ 1529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:04:50.918519 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:04:50.919106 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:04:50.919221 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:04:50.919947 (+ 726us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:04:50.919950 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:04:50.919951 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 09:04:50.920145 (+ 194us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:04:50.920188 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 09:04:50.920202 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:04:50.920268 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0619 09:04:50.920378 (+ 110us) server_negotiation.cc:299] Negotiation successful
0619 09:04:50.920551 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":64,"threads_started":1}
I20260619 09:04:50.921134 20146 catalog_manager.cc:1347] Generated new cluster ID: 7225dc0bc0df494692436d5885421076
I20260619 09:04:50.921188 20146 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 09:04:50.931960 20146 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 09:04:50.932683 20146 catalog_manager.cc:1504] Loading token signing keys...
I20260619 09:04:50.948997 20146 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 131add0a010c4580a9162628fd6d45f0: Generated new TSK 0
I20260619 09:04:50.949239 20146 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 09:04:50.957934 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ez5NB/build/release/bin/kudu
/tmp/dist-test-task4ez5NB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781859658134510-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781859658134510-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781859658134510-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781859658134510-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38129
--builtin_ntp_servers=127.0.98.20:40953
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 09:04:51.040395 20158 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 09:04:51.040588 20158 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 09:04:51.040608 20158 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 09:04:51.041718 20158 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 09:04:51.041757 20158 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 09:04:51.041783 20158 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 09:04:51.043161 20158 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40953
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781859658134510-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781859658134510-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781859658134510-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38129
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20158
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781859658134510-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 09:04:51.043370 20158 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 09:04:51.043646 20158 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 09:04:51.048223 20163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:04:51.048257 20158 server_base.cc:1034] running on GCE node
W20260619 09:04:51.048254 20166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:04:51.048398 20164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:04:51.048607 20158 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 09:04:51.048828 20158 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 09:04:51.049959 20158 hybrid_clock.cc:648] HybridClock initialized: now 1781859891049943 us; error 32 us; skew 500 ppm
I20260619 09:04:51.051090 20158 webserver.cc:460] Webserver started at http://127.0.98.1:46363/ using document root <none> and password file <none>
I20260619 09:04:51.051282 20158 fs_manager.cc:359] Metadata directory not provided
I20260619 09:04:51.051327 20158 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 09:04:51.051448 20158 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 09:04:51.052438 20158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781859658134510-392-0/minicluster-data/ts-0/data/instance:
uuid: "a06c93e8ba184eac8884c4e300c16a23"
format_stamp: "Formatted at 2026-06-19 09:04:51 on dist-test-slave-mtdr"
server_key: "2302134947175597a5be5bcee3fab430"
server_key_iv: "ed06cad2517b1d3fdb99ce882ea51ae9"
server_key_version: "encryptionkey@0"
I20260619 09:04:51.052759 20158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781859658134510-392-0/minicluster-data/ts-0/wal/instance:
uuid: "a06c93e8ba184eac8884c4e300c16a23"
format_stamp: "Formatted at 2026-06-19 09:04:51 on dist-test-slave-mtdr"
server_key: "2302134947175597a5be5bcee3fab430"
server_key_iv: "ed06cad2517b1d3fdb99ce882ea51ae9"
server_key_version: "encryptionkey@0"
I20260619 09:04:51.054018 20158 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 09:04:51.054752 20172 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 09:04:51.054932 20158 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 09:04:51.055007 20158 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781859658134510-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781859658134510-392-0/minicluster-data/ts-0/wal
uuid: "a06c93e8ba184eac8884c4e300c16a23"
format_stamp: "Formatted at 2026-06-19 09:04:51 on dist-test-slave-mtdr"
server_key: "2302134947175597a5be5bcee3fab430"
server_key_iv: "ed06cad2517b1d3fdb99ce882ea51ae9"
server_key_version: "encryptionkey@0"
I20260619 09:04:51.055075 20158 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781859658134510-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781859658134510-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781859658134510-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 09:04:51.079340 20158 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 09:04:51.079615 20158 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 09:04:51.079730 20158 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 09:04:51.079952 20158 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 09:04:51.080261 20158 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 09:04:51.080293 20158 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 09:04:51.080314 20158 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 09:04:51.080336 20158 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 09:04:51.085642 20158 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37037
I20260619 09:04:51.085682 20285 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37037 every 8 connection(s)
I20260619 09:04:51.086022 20158 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781859658134510-392-0/minicluster-data/ts-0/data/info.pb
I20260619 09:04:51.091097 20138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:04:51.086719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47077 (local address 127.0.98.62:38129)
0619 09:04:51.086795 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0619 09:04:51.086797 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:04:51.087039 (+ 242us) server_negotiation.cc:407] Connection header received
0619 09:04:51.087512 (+ 473us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:04:51.087513 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:04:51.087525 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:04:51.087561 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:04:51.088194 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:04:51.088691 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:04:51.089768 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:04:51.089961 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:04:51.090020 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:04:51.090858 (+ 838us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:04:51.090859 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:04:51.090859 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 09:04:51.090864 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:04:51.090880 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 09:04:51.090884 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:04:51.090929 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0619 09:04:51.091015 (+ 86us) server_negotiation.cc:299] Negotiation successful
0619 09:04:51.091040 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260619 09:04:51.091261 20288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:04:51.086704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38129 (local address 127.0.98.1:47077)
0619 09:04:51.086927 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0619 09:04:51.086961 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0619 09:04:51.087427 (+ 466us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:04:51.087620 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:04:51.087623 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:04:51.087632 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0619 09:04:51.088097 (+ 465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:04:51.088102 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:04:51.088793 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:04:51.088794 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:04:51.089617 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:04:51.089620 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:04:51.090054 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:04:51.090056 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:04:51.090167 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:04:51.090692 (+ 525us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 09:04:51.090692 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0619 09:04:51.090720 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 09:04:51.090742 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 09:04:51.090945 (+ 203us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 09:04:51.090949 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 09:04:51.090952 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 09:04:51.091003 (+ 51us) client_negotiation.cc:240] Negotiation successful
0619 09:04:51.091082 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":87,"threads_started":1}
I20260619 09:04:51.091720 20286 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38129
I20260619 09:04:51.091794 20286 heartbeater.cc:464] Registering TS with master...
I20260619 09:04:51.091952 20286 heartbeater.cc:511] Master 127.0.98.62:38129 requested a full tablet report, sending...
I20260619 09:04:51.092411 20098 ts_manager.cc:194] Registered new tserver with Master: a06c93e8ba184eac8884c4e300c16a23 (127.0.98.1:37037)
I20260619 09:04:51.093007 20098 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:47077
I20260619 09:04:51.093845 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ez5NB/build/release/bin/kudu as pid 20158
I20260619 09:04:51.093922 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781859658134510-392-0/minicluster-data/ts-0/wal/instance
I20260619 09:04:51.094079 392 external_mini_cluster.cc:1428] Setting key 092839636d3d7fbd8f9471e4c9d09e1a
I20260619 09:04:51.094900 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ez5NB/build/release/bin/kudu
/tmp/dist-test-task4ez5NB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781859658134510-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781859658134510-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781859658134510-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781859658134510-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38129
--builtin_ntp_servers=127.0.98.20:40953
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 09:04:51.171312 20289 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 09:04:51.171494 20289 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 09:04:51.171514 20289 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 09:04:51.172577 20289 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 09:04:51.172607 20289 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 09:04:51.172629 20289 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 09:04:51.173861 20289 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40953
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781859658134510-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781859658134510-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781859658134510-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38129
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20289
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781859658134510-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 09:04:51.174028 20289 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 09:04:51.174250 20289 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 09:04:51.178853 20295 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:04:51.178843 20294 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:04:51.178845 20297 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:04:51.179001 20289 server_base.cc:1034] running on GCE node
I20260619 09:04:51.179253 20289 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 09:04:51.179476 20289 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 09:04:51.180620 20289 hybrid_clock.cc:648] HybridClock initialized: now 1781859891180604 us; error 30 us; skew 500 ppm
I20260619 09:04:51.181821 20289 webserver.cc:460] Webserver started at http://127.0.98.2:41681/ using document root <none> and password file <none>
I20260619 09:04:51.182042 20289 fs_manager.cc:359] Metadata directory not provided
I20260619 09:04:51.182090 20289 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 09:04:51.182224 20289 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 09:04:51.183259 20289 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781859658134510-392-0/minicluster-data/ts-1/data/instance:
uuid: "116ffbe3bd2741a192c4a7c99d08a924"
format_stamp: "Formatted at 2026-06-19 09:04:51 on dist-test-slave-mtdr"
server_key: "599024c7454d20fa87a7449ab654f659"
server_key_iv: "a1a8b304a52c76f074d1223f0be49200"
server_key_version: "encryptionkey@0"
I20260619 09:04:51.183576 20289 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781859658134510-392-0/minicluster-data/ts-1/wal/instance:
uuid: "116ffbe3bd2741a192c4a7c99d08a924"
format_stamp: "Formatted at 2026-06-19 09:04:51 on dist-test-slave-mtdr"
server_key: "599024c7454d20fa87a7449ab654f659"
server_key_iv: "a1a8b304a52c76f074d1223f0be49200"
server_key_version: "encryptionkey@0"
I20260619 09:04:51.184767 20289 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 09:04:51.185472 20303 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 09:04:51.185626 20289 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 09:04:51.185695 20289 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781859658134510-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781859658134510-392-0/minicluster-data/ts-1/wal
uuid: "116ffbe3bd2741a192c4a7c99d08a924"
format_stamp: "Formatted at 2026-06-19 09:04:51 on dist-test-slave-mtdr"
server_key: "599024c7454d20fa87a7449ab654f659"
server_key_iv: "a1a8b304a52c76f074d1223f0be49200"
server_key_version: "encryptionkey@0"
I20260619 09:04:51.185761 20289 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781859658134510-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781859658134510-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781859658134510-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 09:04:51.194563 20289 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 09:04:51.194832 20289 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 09:04:51.194957 20289 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 09:04:51.195149 20289 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 09:04:51.195430 20289 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 09:04:51.195461 20289 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 09:04:51.195487 20289 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 09:04:51.195511 20289 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 09:04:51.200799 20289 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45237
I20260619 09:04:51.200852 20416 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45237 every 8 connection(s)
I20260619 09:04:51.201119 20289 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781859658134510-392-0/minicluster-data/ts-1/data/info.pb
I20260619 09:04:51.205892 20138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:04:51.201742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41257 (local address 127.0.98.62:38129)
0619 09:04:51.201784 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0619 09:04:51.201786 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:04:51.201991 (+ 205us) server_negotiation.cc:407] Connection header received
0619 09:04:51.202445 (+ 454us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:04:51.202446 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:04:51.202457 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:04:51.202505 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:04:51.203159 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:04:51.203656 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:04:51.204661 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:04:51.204858 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:04:51.204912 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:04:51.205685 (+ 773us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:04:51.205686 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:04:51.205686 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 09:04:51.205690 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:04:51.205705 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 09:04:51.205708 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:04:51.205750 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0619 09:04:51.205807 (+ 57us) server_negotiation.cc:299] Negotiation successful
0619 09:04:51.205830 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260619 09:04:51.206048 20419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:04:51.201728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38129 (local address 127.0.98.2:41257)
0619 09:04:51.201884 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0619 09:04:51.201915 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0619 09:04:51.202358 (+ 443us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:04:51.202561 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:04:51.202565 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:04:51.202572 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0619 09:04:51.203060 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:04:51.203065 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:04:51.203780 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:04:51.203781 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:04:51.204573 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:04:51.204576 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:04:51.204940 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:04:51.204940 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:04:51.205050 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:04:51.205557 (+ 507us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 09:04:51.205557 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0619 09:04:51.205584 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 09:04:51.205605 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 09:04:51.205752 (+ 147us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 09:04:51.205754 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 09:04:51.205756 (+ 2us) client_negotiation.cc:769] Sending connection context
0619 09:04:51.205800 (+ 44us) client_negotiation.cc:240] Negotiation successful
0619 09:04:51.205874 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":81,"threads_started":1}
I20260619 09:04:51.206465 20417 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38129
I20260619 09:04:51.206552 20417 heartbeater.cc:464] Registering TS with master...
I20260619 09:04:51.206717 20417 heartbeater.cc:511] Master 127.0.98.62:38129 requested a full tablet report, sending...
I20260619 09:04:51.207096 20098 ts_manager.cc:194] Registered new tserver with Master: 116ffbe3bd2741a192c4a7c99d08a924 (127.0.98.2:45237)
I20260619 09:04:51.207448 20098 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:41257
I20260619 09:04:51.209096 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ez5NB/build/release/bin/kudu as pid 20289
I20260619 09:04:51.209215 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781859658134510-392-0/minicluster-data/ts-1/wal/instance
I20260619 09:04:51.209394 392 external_mini_cluster.cc:1428] Setting key 73ba0eed6f670ad0ad8d6eb09c7edc73
I20260619 09:04:51.210170 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ez5NB/build/release/bin/kudu
/tmp/dist-test-task4ez5NB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781859658134510-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781859658134510-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781859658134510-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781859658134510-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38129
--builtin_ntp_servers=127.0.98.20:40953
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 09:04:51.287407 20420 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 09:04:51.287603 20420 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 09:04:51.287631 20420 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 09:04:51.288753 20420 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 09:04:51.288790 20420 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 09:04:51.288822 20420 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 09:04:51.290124 20420 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40953
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781859658134510-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781859658134510-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781859658134510-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38129
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20420
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781859658134510-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 09:04:51.290341 20420 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 09:04:51.290601 20420 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 09:04:51.295320 20425 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:04:51.295322 20426 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:04:51.295315 20428 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:04:51.295347 20420 server_base.cc:1034] running on GCE node
I20260619 09:04:51.295722 20420 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 09:04:51.295949 20420 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 09:04:51.297096 20420 hybrid_clock.cc:648] HybridClock initialized: now 1781859891297085 us; error 35 us; skew 500 ppm
I20260619 09:04:51.298357 20420 webserver.cc:460] Webserver started at http://127.0.98.3:38323/ using document root <none> and password file <none>
I20260619 09:04:51.298560 20420 fs_manager.cc:359] Metadata directory not provided
I20260619 09:04:51.298604 20420 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 09:04:51.298691 20420 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 09:04:51.299743 20420 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781859658134510-392-0/minicluster-data/ts-2/data/instance:
uuid: "1d5fb37f11ec44479a1a793e18f9bc71"
format_stamp: "Formatted at 2026-06-19 09:04:51 on dist-test-slave-mtdr"
server_key: "a7bacfca9f188c4ccf7b6de1f23af3e7"
server_key_iv: "261d3137e8a1cba5915da20ef17ee02a"
server_key_version: "encryptionkey@0"
I20260619 09:04:51.300062 20420 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781859658134510-392-0/minicluster-data/ts-2/wal/instance:
uuid: "1d5fb37f11ec44479a1a793e18f9bc71"
format_stamp: "Formatted at 2026-06-19 09:04:51 on dist-test-slave-mtdr"
server_key: "a7bacfca9f188c4ccf7b6de1f23af3e7"
server_key_iv: "261d3137e8a1cba5915da20ef17ee02a"
server_key_version: "encryptionkey@0"
I20260619 09:04:51.301255 20420 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 09:04:51.301936 20434 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 09:04:51.302084 20420 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 09:04:51.302171 20420 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781859658134510-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781859658134510-392-0/minicluster-data/ts-2/wal
uuid: "1d5fb37f11ec44479a1a793e18f9bc71"
format_stamp: "Formatted at 2026-06-19 09:04:51 on dist-test-slave-mtdr"
server_key: "a7bacfca9f188c4ccf7b6de1f23af3e7"
server_key_iv: "261d3137e8a1cba5915da20ef17ee02a"
server_key_version: "encryptionkey@0"
I20260619 09:04:51.302232 20420 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781859658134510-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781859658134510-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781859658134510-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 09:04:51.316393 20420 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 09:04:51.316707 20420 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 09:04:51.316838 20420 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 09:04:51.317049 20420 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 09:04:51.317373 20420 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 09:04:51.317407 20420 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 09:04:51.317435 20420 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 09:04:51.317458 20420 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 09:04:51.323055 20420 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44241
I20260619 09:04:51.323117 20547 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44241 every 8 connection(s)
I20260619 09:04:51.323403 20420 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781859658134510-392-0/minicluster-data/ts-2/data/info.pb
I20260619 09:04:51.324589 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ez5NB/build/release/bin/kudu as pid 20420
I20260619 09:04:51.324697 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781859658134510-392-0/minicluster-data/ts-2/wal/instance
I20260619 09:04:51.324901 392 external_mini_cluster.cc:1428] Setting key 8d90e5e0b532a666e55147cbd810d9cd
I20260619 09:04:51.328899 20138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:04:51.324059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47533 (local address 127.0.98.62:38129)
0619 09:04:51.324137 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0619 09:04:51.324139 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:04:51.324390 (+ 251us) server_negotiation.cc:407] Connection header received
0619 09:04:51.325097 (+ 707us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:04:51.325098 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:04:51.325113 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:04:51.325190 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:04:51.326034 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:04:51.326546 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:04:51.327611 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:04:51.327803 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:04:51.327860 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:04:51.328671 (+ 811us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:04:51.328673 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:04:51.328673 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 09:04:51.328677 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:04:51.328692 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 09:04:51.328696 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:04:51.328746 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0619 09:04:51.328814 (+ 68us) server_negotiation.cc:299] Negotiation successful
0619 09:04:51.328841 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 09:04:51.329073 20550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:04:51.324040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38129 (local address 127.0.98.3:47533)
0619 09:04:51.324248 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0619 09:04:51.324281 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0619 09:04:51.324907 (+ 626us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:04:51.325270 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:04:51.325275 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:04:51.325286 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0619 09:04:51.325920 (+ 634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:04:51.325926 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:04:51.326655 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:04:51.326656 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:04:51.327520 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:04:51.327522 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:04:51.327906 (+ 384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:04:51.327907 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:04:51.328035 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:04:51.328518 (+ 483us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 09:04:51.328519 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 09:04:51.328548 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 09:04:51.328570 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 09:04:51.328754 (+ 184us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 09:04:51.328757 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 09:04:51.328759 (+ 2us) client_negotiation.cc:769] Sending connection context
0619 09:04:51.328808 (+ 49us) client_negotiation.cc:240] Negotiation successful
0619 09:04:51.328887 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":110,"threads_started":1}
I20260619 09:04:51.329536 20548 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38129
I20260619 09:04:51.329623 20548 heartbeater.cc:464] Registering TS with master...
I20260619 09:04:51.329793 20548 heartbeater.cc:511] Master 127.0.98.62:38129 requested a full tablet report, sending...
I20260619 09:04:51.330127 20098 ts_manager.cc:194] Registered new tserver with Master: 1d5fb37f11ec44479a1a793e18f9bc71 (127.0.98.3:44241)
I20260619 09:04:51.330492 20098 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:47533
I20260619 09:04:51.336622 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 09:04:51.340816 20138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:04:51.338019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53362 (local address 127.0.98.62:38129)
0619 09:04:51.338080 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0619 09:04:51.338082 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:04:51.338174 (+ 92us) server_negotiation.cc:407] Connection header received
0619 09:04:51.338277 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:04:51.338278 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:04:51.338289 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:04:51.338348 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:04:51.338503 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:04:51.339044 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:04:51.339764 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:04:51.339961 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:04:51.340013 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:04:51.340598 (+ 585us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:04:51.340599 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:04:51.340600 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 09:04:51.340603 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:04:51.340616 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 09:04:51.340620 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:04:51.340664 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0619 09:04:51.340714 (+ 50us) server_negotiation.cc:299] Negotiation successful
0619 09:04:51.340737 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260619 09:04:51.342392 20098 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53362:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 09:04:51.344043 20098 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 09:04:51.350281 20559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:04:51.346259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37100 (local address 127.0.98.1:37037)
0619 09:04:51.346730 (+ 471us) server_negotiation.cc:206] Beginning negotiation
0619 09:04:51.346734 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 09:04:51.346743 (+ 9us) server_negotiation.cc:407] Connection header received
0619 09:04:51.346774 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:04:51.346776 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:04:51.346800 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:04:51.347018 (+ 218us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 09:04:51.347150 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:04:51.347943 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:04:51.349045 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:04:51.349877 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:04:51.349953 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:04:51.350050 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0619 09:04:51.350139 (+ 89us) server_negotiation.cc:299] Negotiation successful
0619 09:04:51.350182 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":437,"thread_start_us":90,"threads_started":1}
I20260619 09:04:51.350281 20556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:04:51.346161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37037 (local address 127.0.0.1:37100)
0619 09:04:51.346341 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0619 09:04:51.346378 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0619 09:04:51.346438 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:04:51.346920 (+ 482us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:04:51.346921 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:04:51.346923 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 09:04:51.346951 (+ 28us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:04:51.346952 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:04:51.348018 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:04:51.348019 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:04:51.348914 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:04:51.348916 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:04:51.349966 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:04:51.349967 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:04:51.350043 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:04:51.350057 (+ 14us) client_negotiation.cc:769] Sending connection context
0619 09:04:51.350129 (+ 72us) client_negotiation.cc:240] Negotiation successful
0619 09:04:51.350202 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"mutex_wait_us":27,"thread_start_us":73,"threads_started":1}
I20260619 09:04:51.350445 20561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:04:51.346549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42224 (local address 127.0.98.3:44241)
0619 09:04:51.346704 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0619 09:04:51.346708 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 09:04:51.346717 (+ 9us) server_negotiation.cc:407] Connection header received
0619 09:04:51.346744 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:04:51.346745 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:04:51.346757 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:04:51.346868 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 09:04:51.347074 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:04:51.347843 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:04:51.349290 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:04:51.349958 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:04:51.350011 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:04:51.350078 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0619 09:04:51.350324 (+ 246us) server_negotiation.cc:299] Negotiation successful
0619 09:04:51.350360 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":81,"threads_started":1}
I20260619 09:04:51.350503 20560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:04:51.346351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44241 (local address 127.0.0.1:42224)
0619 09:04:51.346457 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0619 09:04:51.346466 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 09:04:51.346506 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:04:51.346879 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:04:51.346883 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:04:51.346886 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 09:04:51.346947 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:04:51.346949 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:04:51.347925 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:04:51.347926 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:04:51.349214 (+ 1288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:04:51.349217 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:04:51.350111 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:04:51.350112 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:04:51.350171 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:04:51.350175 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 09:04:51.350221 (+ 46us) client_negotiation.cc:240] Negotiation successful
0619 09:04:51.350261 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":92,"mutex_wait_us":23,"thread_start_us":70,"threads_started":1}
I20260619 09:04:51.351387 20557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:04:51.346161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45237 (local address 127.0.0.1:55156)
0619 09:04:51.346440 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0619 09:04:51.346451 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 09:04:51.346507 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:04:51.348278 (+ 1771us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:04:51.348278 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:04:51.348281 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 09:04:51.348317 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:04:51.348318 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:04:51.349500 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:04:51.349502 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:04:51.350303 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:04:51.350305 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:04:51.351171 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:04:51.351172 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:04:51.351233 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:04:51.351236 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 09:04:51.351281 (+ 45us) client_negotiation.cc:240] Negotiation successful
0619 09:04:51.351318 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"mutex_wait_us":5,"thread_start_us":62,"threads_started":1}
I20260619 09:04:51.351410 20558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:04:51.346260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55156 (local address 127.0.98.2:45237)
0619 09:04:51.348096 (+ 1836us) server_negotiation.cc:206] Beginning negotiation
0619 09:04:51.348101 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 09:04:51.348110 (+ 9us) server_negotiation.cc:407] Connection header received
0619 09:04:51.348147 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:04:51.348149 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:04:51.348164 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:04:51.348242 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 09:04:51.348394 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:04:51.349433 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:04:51.350392 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:04:51.351090 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:04:51.351155 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:04:51.351240 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0619 09:04:51.351296 (+ 56us) server_negotiation.cc:299] Negotiation successful
0619 09:04:51.351332 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1795,"thread_start_us":93,"threads_started":1}
I20260619 09:04:51.351302 20482 tablet_service.cc:1467] Processing CreateTablet for tablet 4452b3635e9b4f15a8b3d5259cfc4ed1 (DEFAULT_TABLE table=test-table [id=6e3c91e55d3e40e9b5bf6ec6e171c38d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 09:04:51.351287 20220 tablet_service.cc:1467] Processing CreateTablet for tablet 4452b3635e9b4f15a8b3d5259cfc4ed1 (DEFAULT_TABLE table=test-table [id=6e3c91e55d3e40e9b5bf6ec6e171c38d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 09:04:51.351624 20482 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4452b3635e9b4f15a8b3d5259cfc4ed1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 09:04:51.351624 20220 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4452b3635e9b4f15a8b3d5259cfc4ed1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 09:04:51.352061 20351 tablet_service.cc:1467] Processing CreateTablet for tablet 4452b3635e9b4f15a8b3d5259cfc4ed1 (DEFAULT_TABLE table=test-table [id=6e3c91e55d3e40e9b5bf6ec6e171c38d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 09:04:51.352314 20351 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4452b3635e9b4f15a8b3d5259cfc4ed1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 09:04:51.354569 20562 tablet_bootstrap.cc:492] T 4452b3635e9b4f15a8b3d5259cfc4ed1 P 116ffbe3bd2741a192c4a7c99d08a924: Bootstrap starting.
I20260619 09:04:51.355011 20564 tablet_bootstrap.cc:492] T 4452b3635e9b4f15a8b3d5259cfc4ed1 P a06c93e8ba184eac8884c4e300c16a23: Bootstrap starting.
I20260619 09:04:51.355247 20562 tablet_bootstrap.cc:654] T 4452b3635e9b4f15a8b3d5259cfc4ed1 P 116ffbe3bd2741a192c4a7c99d08a924: Neither blocks nor log segments found. Creating new log.
I20260619 09:04:51.355280 20563 tablet_bootstrap.cc:492] T 4452b3635e9b4f15a8b3d5259cfc4ed1 P 1d5fb37f11ec44479a1a793e18f9bc71: Bootstrap starting.
I20260619 09:04:51.355521 20562 log.cc:824] T 4452b3635e9b4f15a8b3d5259cfc4ed1 P 116ffbe3bd2741a192c4a7c99d08a924: Log is configured to *not* fsync() on all Append() calls
I20260619 09:04:51.355639 20564 tablet_bootstrap.cc:654] T 4452b3635e9b4f15a8b3d5259cfc4ed1 P a06c93e8ba184eac8884c4e300c16a23: Neither blocks nor log segments found. Creating new log.
I20260619 09:04:51.355867 20563 tablet_bootstrap.cc:654] T 4452b3635e9b4f15a8b3d5259cfc4ed1 P 1d5fb37f11ec44479a1a793e18f9bc71: Neither blocks nor log segments found. Creating new log.
I20260619 09:04:51.355958 20564 log.cc:824] T 4452b3635e9b4f15a8b3d5259cfc4ed1 P a06c93e8ba184eac8884c4e300c16a23: Log is configured to *not* fsync() on all Append() calls
I20260619 09:04:51.356149 20563 log.cc:824] T 4452b3635e9b4f15a8b3d5259cfc4ed1 P 1d5fb37f11ec44479a1a793e18f9bc71: Log is configured to *not* fsync() on all Append() calls
I20260619 09:04:51.356189 20562 tablet_bootstrap.cc:492] T 4452b3635e9b4f15a8b3d5259cfc4ed1 P 116ffbe3bd2741a192c4a7c99d08a924: No bootstrap required, opened a new log
I20260619 09:04:51.356247 20562 ts_tablet_manager.cc:1397] T 4452b3635e9b4f15a8b3d5259cfc4ed1 P 116ffbe3bd2741a192c4a7c99d08a924: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260619 09:04:51.356663 20564 tablet_bootstrap.cc:492] T 4452b3635e9b4f15a8b3d5259cfc4ed1 P a06c93e8ba184eac8884c4e300c16a23: No bootstrap required, opened a new log
I20260619 09:04:51.356720 20564 ts_tablet_manager.cc:1397] T 4452b3635e9b4f15a8b3d5259cfc4ed1 P a06c93e8ba184eac8884c4e300c16a23: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260619 09:04:51.358279 20562 raft_consensus.cc:348] T 4452b3635e9b4f15a8b3d5259cfc4ed1 P 116ffbe3bd2741a192c4a7c99d08a924 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "116ffbe3bd2741a192c4a7c99d08a924" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45237 } } peers { permanent_uuid: "1d5fb37f11ec44479a1a793e18f9bc71" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44241 } } peers { permanent_uuid: "a06c93e8ba184eac8884c4e300c16a23" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37037 } }
I20260619 09:04:51.358408 20562 raft_consensus.cc:374] T 4452b3635e9b4f15a8b3d5259cfc4ed1 P 116ffbe3bd2741a192c4a7c99d08a924 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 09:04:51.358429 20562 raft_consensus.cc:729] T 4452b3635e9b4f15a8b3d5259cfc4ed1 P 116ffbe3bd2741a192c4a7c99d08a924 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 116ffbe3bd2741a192c4a7c99d08a924, State: Initialized, Role: FOLLOWER
I20260619 09:04:51.358525 20562 consensus_queue.cc:260] T 4452b3635e9b4f15a8b3d5259cfc4ed1 P 116ffbe3bd2741a192c4a7c99d08a924 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "116ffbe3bd2741a192c4a7c99d08a924" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45237 } } peers { permanent_uuid: "1d5fb37f11ec44479a1a793e18f9bc71" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44241 } } peers { permanent_uuid: "a06c93e8ba184eac8884c4e300c16a23" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37037 } }
I20260619 09:04:51.358774 20562 ts_tablet_manager.cc:1428] T 4452b3635e9b4f15a8b3d5259cfc4ed1 P 116ffbe3bd2741a192c4a7c99d08a924: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260619 09:04:51.358665 20564 raft_consensus.cc:348] T 4452b3635e9b4f15a8b3d5259cfc4ed1 P a06c93e8ba184eac8884c4e300c16a23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "116ffbe3bd2741a192c4a7c99d08a924" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45237 } } peers { permanent_uuid: "1d5fb37f11ec44479a1a793e18f9bc71" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44241 } } peers { permanent_uuid: "a06c93e8ba184eac8884c4e300c16a23" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37037 } }
I20260619 09:04:51.358846 20417 heartbeater.cc:503] Master 127.0.98.62:38129 was elected leader, sending a full tablet report...
I20260619 09:04:51.358850 20564 raft_consensus.cc:374] T 4452b3635e9b4f15a8b3d5259cfc4ed1 P a06c93e8ba184eac8884c4e300c16a23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 09:04:51.358889 20564 raft_consensus.cc:729] T 4452b3635e9b4f15a8b3d5259cfc4ed1 P a06c93e8ba184eac8884c4e300c16a23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a06c93e8ba184eac8884c4e300c16a23, State: Initialized, Role: FOLLOWER
I20260619 09:04:51.359071 20564 consensus_queue.cc:260] T 4452b3635e9b4f15a8b3d5259cfc4ed1 P a06c93e8ba184eac8884c4e300c16a23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "116ffbe3bd2741a192c4a7c99d08a924" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45237 } } peers { permanent_uuid: "1d5fb37f11ec44479a1a793e18f9bc71" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44241 } } peers { permanent_uuid: "a06c93e8ba184eac8884c4e300c16a23" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37037 } }
I20260619 09:04:51.359232 20569 raft_consensus.cc:482] T 4452b3635e9b4f15a8b3d5259cfc4ed1 P a06c93e8ba184eac8884c4e300c16a23 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 09:04:51.359316 20564 ts_tablet_manager.cc:1428] T 4452b3635e9b4f15a8b3d5259cfc4ed1 P a06c93e8ba184eac8884c4e300c16a23: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260619 09:04:51.359347 20569 raft_consensus.cc:504] T 4452b3635e9b4f15a8b3d5259cfc4ed1 P a06c93e8ba184eac8884c4e300c16a23 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "116ffbe3bd2741a192c4a7c99d08a924" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45237 } } peers { permanent_uuid: "1d5fb37f11ec44479a1a793e18f9bc71" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44241 } } peers { permanent_uuid: "a06c93e8ba184eac8884c4e300c16a23" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37037 } }
I20260619 09:04:51.359515 20286 heartbeater.cc:503] Master 127.0.98.62:38129 was elected leader, sending a full tablet report...
I20260619 09:04:51.359637 20569 leader_election.cc:288] T 4452b3635e9b4f15a8b3d5259cfc4ed1 P a06c93e8ba184eac8884c4e300c16a23 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 116ffbe3bd2741a192c4a7c99d08a924 (127.0.98.2:45237), 1d5fb37f11ec44479a1a793e18f9bc71 (127.0.98.3:44241)
I20260619 09:04:51.359694 20563 tablet_bootstrap.cc:492] T 4452b3635e9b4f15a8b3d5259cfc4ed1 P 1d5fb37f11ec44479a1a793e18f9bc71: No bootstrap required, opened a new log
I20260619 09:04:51.359755 20563 ts_tablet_manager.cc:1397] T 4452b3635e9b4f15a8b3d5259cfc4ed1 P 1d5fb37f11ec44479a1a793e18f9bc71: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260619 09:04:51.362166 20563 raft_consensus.cc:348] T 4452b3635e9b4f15a8b3d5259cfc4ed1 P 1d5fb37f11ec44479a1a793e18f9bc71 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "116ffbe3bd2741a192c4a7c99d08a924" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45237 } } peers { permanent_uuid: "1d5fb37f11ec44479a1a793e18f9bc71" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44241 } } peers { permanent_uuid: "a06c93e8ba184eac8884c4e300c16a23" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37037 } }
I20260619 09:04:51.362322 20563 raft_consensus.cc:374] T 4452b3635e9b4f15a8b3d5259cfc4ed1 P 1d5fb37f11ec44479a1a793e18f9bc71 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 09:04:51.362399 20563 raft_consensus.cc:729] T 4452b3635e9b4f15a8b3d5259cfc4ed1 P 1d5fb37f11ec44479a1a793e18f9bc71 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d5fb37f11ec44479a1a793e18f9bc71, State: Initialized, Role: FOLLOWER
I20260619 09:04:51.362527 20563 consensus_queue.cc:260] T 4452b3635e9b4f15a8b3d5259cfc4ed1 P 1d5fb37f11ec44479a1a793e18f9bc71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "116ffbe3bd2741a192c4a7c99d08a924" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45237 } } peers { permanent_uuid: "1d5fb37f11ec44479a1a793e18f9bc71" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44241 } } peers { permanent_uuid: "a06c93e8ba184eac8884c4e300c16a23" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37037 } }
I20260619 09:04:51.362776 20563 ts_tablet_manager.cc:1428] T 4452b3635e9b4f15a8b3d5259cfc4ed1 P 1d5fb37f11ec44479a1a793e18f9bc71: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260619 09:04:51.362802 20548 heartbeater.cc:503] Master 127.0.98.62:38129 was elected leader, sending a full tablet report...
I20260619 09:04:51.363186 20571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:04:51.359885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44241 (local address 127.0.98.1:35331)
0619 09:04:51.360298 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0619 09:04:51.360308 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 09:04:51.360383 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:04:51.360553 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:04:51.360554 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:04:51.360555 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 09:04:51.360642 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:04:51.360644 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:04:51.361309 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:04:51.361310 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:04:51.362230 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:04:51.362233 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:04:51.362932 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:04:51.362934 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:04:51.363016 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:04:51.363020 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 09:04:51.363069 (+ 49us) client_negotiation.cc:240] Negotiation successful
0619 09:04:51.363106 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":68,"threads_started":1}
I20260619 09:04:51.363230 20558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:04:51.359807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60403 (local address 127.0.98.2:45237)
0619 09:04:51.359846 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0619 09:04:51.359847 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 09:04:51.360065 (+ 218us) server_negotiation.cc:407] Connection header received
0619 09:04:51.360130 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:04:51.360131 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:04:51.360140 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:04:51.360200 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 09:04:51.360397 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:04:51.361071 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:04:51.362389 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:04:51.362910 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:04:51.362972 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:04:51.362981 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0619 09:04:51.363137 (+ 156us) server_negotiation.cc:299] Negotiation successful
0619 09:04:51.363171 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260619 09:04:51.363273 20288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:04:51.359773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45237 (local address 127.0.98.1:60403)
0619 09:04:51.359989 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0619 09:04:51.359999 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 09:04:51.360077 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:04:51.360257 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:04:51.360259 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:04:51.360261 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 09:04:51.360325 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:04:51.360327 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:04:51.361161 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:04:51.361162 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:04:51.362295 (+ 1133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:04:51.362298 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:04:51.363012 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:04:51.363013 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:04:51.363079 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:04:51.363082 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 09:04:51.363130 (+ 48us) client_negotiation.cc:240] Negotiation successful
0619 09:04:51.363171 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":200,"mutex_wait_us":24}
I20260619 09:04:51.363478 20561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:04:51.359937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35331 (local address 127.0.98.3:44241)
0619 09:04:51.359969 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0619 09:04:51.359971 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:04:51.360358 (+ 387us) server_negotiation.cc:407] Connection header received
0619 09:04:51.360451 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:04:51.360452 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:04:51.360461 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:04:51.360487 (+ 26us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 09:04:51.360710 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:04:51.361247 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:04:51.362315 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:04:51.362804 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:04:51.362888 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:04:51.362899 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0619 09:04:51.363375 (+ 476us) server_negotiation.cc:299] Negotiation successful
0619 09:04:51.363415 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260619 09:04:51.363492 20371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4452b3635e9b4f15a8b3d5259cfc4ed1" candidate_uuid: "a06c93e8ba184eac8884c4e300c16a23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "116ffbe3bd2741a192c4a7c99d08a924" is_pre_election: true
I20260619 09:04:51.363610 20371 raft_consensus.cc:2454] T 4452b3635e9b4f15a8b3d5259cfc4ed1 P 116ffbe3bd2741a192c4a7c99d08a924 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a06c93e8ba184eac8884c4e300c16a23 in term 0.
I20260619 09:04:51.363821 20502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4452b3635e9b4f15a8b3d5259cfc4ed1" candidate_uuid: "a06c93e8ba184eac8884c4e300c16a23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d5fb37f11ec44479a1a793e18f9bc71" is_pre_election: true
I20260619 09:04:51.363894 20502 raft_consensus.cc:2454] T 4452b3635e9b4f15a8b3d5259cfc4ed1 P 1d5fb37f11ec44479a1a793e18f9bc71 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a06c93e8ba184eac8884c4e300c16a23 in term 0.
I20260619 09:04:51.364068 20174 leader_election.cc:302] T 4452b3635e9b4f15a8b3d5259cfc4ed1 P a06c93e8ba184eac8884c4e300c16a23 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d5fb37f11ec44479a1a793e18f9bc71, a06c93e8ba184eac8884c4e300c16a23; no voters:
I20260619 09:04:51.364187 20569 raft_consensus.cc:2781] T 4452b3635e9b4f15a8b3d5259cfc4ed1 P a06c93e8ba184eac8884c4e300c16a23 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 09:04:51.364251 20569 raft_consensus.cc:482] T 4452b3635e9b4f15a8b3d5259cfc4ed1 P a06c93e8ba184eac8884c4e300c16a23 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 09:04:51.364270 20569 raft_consensus.cc:3037] T 4452b3635e9b4f15a8b3d5259cfc4ed1 P a06c93e8ba184eac8884c4e300c16a23 [term 0 FOLLOWER]: Advancing to term 1
I20260619 09:04:51.364881 20569 raft_consensus.cc:504] T 4452b3635e9b4f15a8b3d5259cfc4ed1 P a06c93e8ba184eac8884c4e300c16a23 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "116ffbe3bd2741a192c4a7c99d08a924" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45237 } } peers { permanent_uuid: "1d5fb37f11ec44479a1a793e18f9bc71" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44241 } } peers { permanent_uuid: "a06c93e8ba184eac8884c4e300c16a23" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37037 } }
I20260619 09:04:51.364986 20569 leader_election.cc:288] T 4452b3635e9b4f15a8b3d5259cfc4ed1 P a06c93e8ba184eac8884c4e300c16a23 [CANDIDATE]: Term 1 election: Requested vote from peers 116ffbe3bd2741a192c4a7c99d08a924 (127.0.98.2:45237), 1d5fb37f11ec44479a1a793e18f9bc71 (127.0.98.3:44241)
I20260619 09:04:51.365144 20502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4452b3635e9b4f15a8b3d5259cfc4ed1" candidate_uuid: "a06c93e8ba184eac8884c4e300c16a23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d5fb37f11ec44479a1a793e18f9bc71"
I20260619 09:04:51.365173 20371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4452b3635e9b4f15a8b3d5259cfc4ed1" candidate_uuid: "a06c93e8ba184eac8884c4e300c16a23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "116ffbe3bd2741a192c4a7c99d08a924"
I20260619 09:04:51.365202 20502 raft_consensus.cc:3037] T 4452b3635e9b4f15a8b3d5259cfc4ed1 P 1d5fb37f11ec44479a1a793e18f9bc71 [term 0 FOLLOWER]: Advancing to term 1
I20260619 09:04:51.365232 20371 raft_consensus.cc:3037] T 4452b3635e9b4f15a8b3d5259cfc4ed1 P 116ffbe3bd2741a192c4a7c99d08a924 [term 0 FOLLOWER]: Advancing to term 1
I20260619 09:04:51.365931 20502 raft_consensus.cc:2454] T 4452b3635e9b4f15a8b3d5259cfc4ed1 P 1d5fb37f11ec44479a1a793e18f9bc71 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a06c93e8ba184eac8884c4e300c16a23 in term 1.
I20260619 09:04:51.365964 20371 raft_consensus.cc:2454] T 4452b3635e9b4f15a8b3d5259cfc4ed1 P 116ffbe3bd2741a192c4a7c99d08a924 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a06c93e8ba184eac8884c4e300c16a23 in term 1.
I20260619 09:04:51.366077 20174 leader_election.cc:302] T 4452b3635e9b4f15a8b3d5259cfc4ed1 P a06c93e8ba184eac8884c4e300c16a23 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 116ffbe3bd2741a192c4a7c99d08a924, a06c93e8ba184eac8884c4e300c16a23; no voters:
I20260619 09:04:51.366153 20569 raft_consensus.cc:2781] T 4452b3635e9b4f15a8b3d5259cfc4ed1 P a06c93e8ba184eac8884c4e300c16a23 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 09:04:51.366225 20569 raft_consensus.cc:686] T 4452b3635e9b4f15a8b3d5259cfc4ed1 P a06c93e8ba184eac8884c4e300c16a23 [term 1 LEADER]: Becoming Leader. State: Replica: a06c93e8ba184eac8884c4e300c16a23, State: Running, Role: LEADER
I20260619 09:04:51.366302 20569 consensus_queue.cc:237] T 4452b3635e9b4f15a8b3d5259cfc4ed1 P a06c93e8ba184eac8884c4e300c16a23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "116ffbe3bd2741a192c4a7c99d08a924" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45237 } } peers { permanent_uuid: "1d5fb37f11ec44479a1a793e18f9bc71" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44241 } } peers { permanent_uuid: "a06c93e8ba184eac8884c4e300c16a23" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37037 } }
I20260619 09:04:51.367024 20098 catalog_manager.cc:5495] T 4452b3635e9b4f15a8b3d5259cfc4ed1 P a06c93e8ba184eac8884c4e300c16a23 reported cstate change: term changed from 0 to 1, leader changed from <none> to a06c93e8ba184eac8884c4e300c16a23 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "a06c93e8ba184eac8884c4e300c16a23" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "116ffbe3bd2741a192c4a7c99d08a924" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1d5fb37f11ec44479a1a793e18f9bc71" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44241 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a06c93e8ba184eac8884c4e300c16a23" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37037 } health_report { overall_health: HEALTHY } } }
I20260619 09:04:51.374357 20559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:04:51.372199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37106 (local address 127.0.98.1:37037)
0619 09:04:51.372238 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0619 09:04:51.372239 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 09:04:51.372273 (+ 34us) server_negotiation.cc:407] Connection header received
0619 09:04:51.372367 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:04:51.372367 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:04:51.372382 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:04:51.372434 (+ 52us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 09:04:51.372565 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:04:51.373034 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:04:51.373756 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:04:51.373937 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:04:51.373980 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:04:51.374076 (+ 96us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 09:04:51.374158 (+ 82us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 09:04:51.374210 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0619 09:04:51.374278 (+ 68us) server_negotiation.cc:299] Negotiation successful
0619 09:04:51.374293 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260619 09:04:51.375880 20502 raft_consensus.cc:1261] T 4452b3635e9b4f15a8b3d5259cfc4ed1 P 1d5fb37f11ec44479a1a793e18f9bc71 [term 1 FOLLOWER]: Refusing update from remote peer a06c93e8ba184eac8884c4e300c16a23: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 09:04:51.375911 20371 raft_consensus.cc:1261] T 4452b3635e9b4f15a8b3d5259cfc4ed1 P 116ffbe3bd2741a192c4a7c99d08a924 [term 1 FOLLOWER]: Refusing update from remote peer a06c93e8ba184eac8884c4e300c16a23: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 09:04:51.376187 20569 consensus_queue.cc:1035] T 4452b3635e9b4f15a8b3d5259cfc4ed1 P a06c93e8ba184eac8884c4e300c16a23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d5fb37f11ec44479a1a793e18f9bc71" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44241 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 09:04:51.376291 20570 consensus_queue.cc:1035] T 4452b3635e9b4f15a8b3d5259cfc4ed1 P a06c93e8ba184eac8884c4e300c16a23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "116ffbe3bd2741a192c4a7c99d08a924" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 09:04:51.379236 20576 mvcc.cc:205] Tried to move back new op lower bound from 7298498115074072576 to 7298498115037450240. Current Snapshot: MvccSnapshot[applied={T|T < 7298498115074072576}]
I20260619 09:04:51.379451 20577 mvcc.cc:205] Tried to move back new op lower bound from 7298498115074072576 to 7298498115037450240. Current Snapshot: MvccSnapshot[applied={T|T < 7298498115074072576}]
I20260619 09:04:51.381107 20097 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:53362:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 09:04:51.381191 20097 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:53362:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 09:04:51.381920 20097 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 131add0a010c4580a9162628fd6d45f0: Sending DeleteTablet for 3 replicas of tablet 4452b3635e9b4f15a8b3d5259cfc4ed1
I20260619 09:04:51.382211 20220 tablet_service.cc:1514] Processing DeleteTablet for tablet 4452b3635e9b4f15a8b3d5259cfc4ed1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 09:04:51 UTC) from {username='slave'} at 127.0.0.1:37100
I20260619 09:04:51.382411 20482 tablet_service.cc:1514] Processing DeleteTablet for tablet 4452b3635e9b4f15a8b3d5259cfc4ed1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 09:04:51 UTC) from {username='slave'} at 127.0.0.1:42224
I20260619 09:04:51.382488 20584 tablet_replica.cc:330] T 4452b3635e9b4f15a8b3d5259cfc4ed1 P a06c93e8ba184eac8884c4e300c16a23: stopping tablet replica
I20260619 09:04:51.382517 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ez5NB/build/release/bin/kudu with pid 20158
I20260619 09:04:51.382570 20584 raft_consensus.cc:2229] T 4452b3635e9b4f15a8b3d5259cfc4ed1 P a06c93e8ba184eac8884c4e300c16a23 [term 1 LEADER]: Raft consensus shutting down.
I20260619 09:04:51.382675 20584 raft_consensus.cc:2258] T 4452b3635e9b4f15a8b3d5259cfc4ed1 P a06c93e8ba184eac8884c4e300c16a23 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 09:04:51.383410 20351 tablet_service.cc:1514] Processing DeleteTablet for tablet 4452b3635e9b4f15a8b3d5259cfc4ed1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 09:04:51 UTC) from {username='slave'} at 127.0.0.1:55156
I20260619 09:04:51.383844 20585 tablet_replica.cc:330] T 4452b3635e9b4f15a8b3d5259cfc4ed1 P 1d5fb37f11ec44479a1a793e18f9bc71: stopping tablet replica
I20260619 09:04:51.383942 20585 raft_consensus.cc:2229] T 4452b3635e9b4f15a8b3d5259cfc4ed1 P 1d5fb37f11ec44479a1a793e18f9bc71 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 09:04:51.384033 20585 raft_consensus.cc:2258] T 4452b3635e9b4f15a8b3d5259cfc4ed1 P 1d5fb37f11ec44479a1a793e18f9bc71 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 09:04:51.384443 20586 tablet_replica.cc:330] T 4452b3635e9b4f15a8b3d5259cfc4ed1 P 116ffbe3bd2741a192c4a7c99d08a924: stopping tablet replica
I20260619 09:04:51.384446 20585 ts_tablet_manager.cc:1905] T 4452b3635e9b4f15a8b3d5259cfc4ed1 P 1d5fb37f11ec44479a1a793e18f9bc71: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 09:04:51.384538 20586 raft_consensus.cc:2229] T 4452b3635e9b4f15a8b3d5259cfc4ed1 P 116ffbe3bd2741a192c4a7c99d08a924 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 09:04:51.384634 20586 raft_consensus.cc:2258] T 4452b3635e9b4f15a8b3d5259cfc4ed1 P 116ffbe3bd2741a192c4a7c99d08a924 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 09:04:51.385001 20586 ts_tablet_manager.cc:1905] T 4452b3635e9b4f15a8b3d5259cfc4ed1 P 116ffbe3bd2741a192c4a7c99d08a924: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 09:04:51.385973 20585 ts_tablet_manager.cc:1918] T 4452b3635e9b4f15a8b3d5259cfc4ed1 P 1d5fb37f11ec44479a1a793e18f9bc71: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 09:04:51.386041 20585 log.cc:1196] T 4452b3635e9b4f15a8b3d5259cfc4ed1 P 1d5fb37f11ec44479a1a793e18f9bc71: Deleting WAL directory at /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781859658134510-392-0/minicluster-data/ts-2/wal/wals/4452b3635e9b4f15a8b3d5259cfc4ed1
I20260619 09:04:51.386196 20586 ts_tablet_manager.cc:1918] T 4452b3635e9b4f15a8b3d5259cfc4ed1 P 116ffbe3bd2741a192c4a7c99d08a924: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 09:04:51.386264 20586 log.cc:1196] T 4452b3635e9b4f15a8b3d5259cfc4ed1 P 116ffbe3bd2741a192c4a7c99d08a924: Deleting WAL directory at /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781859658134510-392-0/minicluster-data/ts-1/wal/wals/4452b3635e9b4f15a8b3d5259cfc4ed1
I20260619 09:04:51.386265 20585 ts_tablet_manager.cc:1939] T 4452b3635e9b4f15a8b3d5259cfc4ed1 P 1d5fb37f11ec44479a1a793e18f9bc71: Deleting consensus metadata
I20260619 09:04:51.386515 20586 ts_tablet_manager.cc:1939] T 4452b3635e9b4f15a8b3d5259cfc4ed1 P 116ffbe3bd2741a192c4a7c99d08a924: Deleting consensus metadata
I20260619 09:04:51.386684 20083 catalog_manager.cc:4841] TS 1d5fb37f11ec44479a1a793e18f9bc71 (127.0.98.3:44241): tablet 4452b3635e9b4f15a8b3d5259cfc4ed1 (table test-table [id=6e3c91e55d3e40e9b5bf6ec6e171c38d]) successfully deleted
I20260619 09:04:51.386850 20083 catalog_manager.cc:4841] TS 116ffbe3bd2741a192c4a7c99d08a924 (127.0.98.2:45237): tablet 4452b3635e9b4f15a8b3d5259cfc4ed1 (table test-table [id=6e3c91e55d3e40e9b5bf6ec6e171c38d]) successfully deleted
W20260619 09:04:51.387953 20082 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:37037 (error 108)
I20260619 09:04:51.388396 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ez5NB/build/release/bin/kudu with pid 20289
I20260619 09:04:51.388427 20557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:04:51.388280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37037 (local address 127.0.0.1:37110)
0619 09:04:51.388337 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0619 09:04:51.388380 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260619 09:04:51.390259 20082 catalog_manager.cc:4584] TS a06c93e8ba184eac8884c4e300c16a23 (127.0.98.1:37037): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4452b3635e9b4f15a8b3d5259cfc4ed1: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37037: connect: Connection refused (error 111)
I20260619 09:04:51.392977 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ez5NB/build/release/bin/kudu with pid 20420
I20260619 09:04:51.396785 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ez5NB/build/release/bin/kudu with pid 20067
2026-06-19T09:04:51Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (630 ms)
[----------] 6 tests from EncryptionPolicyTest (7880 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781859658134510-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 09:04:51 dist-test-slave-mtdr krb5kdc[20590](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 09:04:51 dist-test-slave-mtdr krb5kdc[20590](info): set up 2 sockets
Jun 19 09:04:51 dist-test-slave-mtdr krb5kdc[20590](info): commencing operation
krb5kdc: starting...
W20260619 09:04:52.342530 392 mini_kdc.cc:121] Time spent starting KDC: real 0.928s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781859658134510-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781859658134510-392-0/krb5kdc/test-user.keytab.
Jun 19 09:04:52 dist-test-slave-mtdr krb5kdc[20590](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859892, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T09:04:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T09:04:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781859658134510-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781859658134510-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 09:04:52.451869 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ez5NB/build/release/bin/kudu
/tmp/dist-test-task4ez5NB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781859658134510-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781859658134510-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781859658134510-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781859658134510-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33343
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781859658134510-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33343
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781859658134510-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781859658134510-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781859658134510-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 09:04:52.529032 20603 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 09:04:52.529228 20603 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 09:04:52.529249 20603 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 09:04:52.530299 20603 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 09:04:52.530329 20603 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 09:04:52.530341 20603 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 09:04:52.530352 20603 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 09:04:52.530364 20603 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 09:04:52.531759 20603 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781859658134510-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781859658134510-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33343
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781859658134510-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33343
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781859658134510-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20603
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781859658134510-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 09:04:52.531940 20603 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 09:04:52.532095 20603 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 09:04:52.536705 20609 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:04:52.536728 20608 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:04:52.536777 20611 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:04:52.536850 20603 server_base.cc:1034] running on GCE node
I20260619 09:04:52.537053 20603 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 09:04:52.537310 20603 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 09:04:52.538443 20603 hybrid_clock.cc:648] HybridClock initialized: now 1781859892538432 us; error 26 us; skew 500 ppm
Jun 19 09:04:52 dist-test-slave-mtdr krb5kdc[20590](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859892, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 09:04:52.543817 20603 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 09:04:52.544227 20603 webserver.cc:460] Webserver started at http://127.0.0.1:35993/ using document root <none> and password file <none>
I20260619 09:04:52.544420 20603 fs_manager.cc:359] Metadata directory not provided
I20260619 09:04:52.544466 20603 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 09:04:52.544587 20603 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 09:04:52.545558 20603 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781859658134510-392-0/minicluster-data/master-0/data/instance:
uuid: "37af3c32820d439fa711652c987de0ef"
format_stamp: "Formatted at 2026-06-19 09:04:52 on dist-test-slave-mtdr"
server_key: "ccba6fc2c6bf1dd40c116d8360037ee8"
server_key_iv: "fdd5eeacd15f29437e9805347137e52a"
server_key_version: "encryptionkey@0"
I20260619 09:04:52.545890 20603 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781859658134510-392-0/minicluster-data/master-0/wal/instance:
uuid: "37af3c32820d439fa711652c987de0ef"
format_stamp: "Formatted at 2026-06-19 09:04:52 on dist-test-slave-mtdr"
server_key: "ccba6fc2c6bf1dd40c116d8360037ee8"
server_key_iv: "fdd5eeacd15f29437e9805347137e52a"
server_key_version: "encryptionkey@0"
I20260619 09:04:52.547164 20603 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 09:04:52.547881 20618 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 09:04:52.548041 20603 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 09:04:52.548120 20603 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781859658134510-392-0/minicluster-data/master-0/data,/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781859658134510-392-0/minicluster-data/master-0/wal
uuid: "37af3c32820d439fa711652c987de0ef"
format_stamp: "Formatted at 2026-06-19 09:04:52 on dist-test-slave-mtdr"
server_key: "ccba6fc2c6bf1dd40c116d8360037ee8"
server_key_iv: "fdd5eeacd15f29437e9805347137e52a"
server_key_version: "encryptionkey@0"
I20260619 09:04:52.548179 20603 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781859658134510-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781859658134510-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781859658134510-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 09:04:52.568393 20603 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 09:04:52.570504 20603 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 09:04:52.570636 20603 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 09:04:52.574405 20603 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33343
I20260619 09:04:52.574450 20670 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33343 every 8 connection(s)
I20260619 09:04:52.574759 20603 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781859658134510-392-0/minicluster-data/master-0/data/info.pb
I20260619 09:04:52.575601 20671 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 09:04:52.576246 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ez5NB/build/release/bin/kudu as pid 20603
I20260619 09:04:52.576376 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781859658134510-392-0/minicluster-data/master-0/wal/instance
I20260619 09:04:52.576592 392 external_mini_cluster.cc:1428] Setting key e69045e8ec9537fe263b47a94a2954c2
I20260619 09:04:52.578541 20671 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 37af3c32820d439fa711652c987de0ef: Bootstrap starting.
I20260619 09:04:52.579160 20671 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 37af3c32820d439fa711652c987de0ef: Neither blocks nor log segments found. Creating new log.
I20260619 09:04:52.579432 20671 log.cc:824] T 00000000000000000000000000000000 P 37af3c32820d439fa711652c987de0ef: Log is configured to *not* fsync() on all Append() calls
I20260619 09:04:52.580083 20671 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 37af3c32820d439fa711652c987de0ef: No bootstrap required, opened a new log
I20260619 09:04:52.582012 20671 raft_consensus.cc:348] T 00000000000000000000000000000000 P 37af3c32820d439fa711652c987de0ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37af3c32820d439fa711652c987de0ef" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33343 } }
I20260619 09:04:52.582160 20671 raft_consensus.cc:374] T 00000000000000000000000000000000 P 37af3c32820d439fa711652c987de0ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 09:04:52.582188 20671 raft_consensus.cc:729] T 00000000000000000000000000000000 P 37af3c32820d439fa711652c987de0ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37af3c32820d439fa711652c987de0ef, State: Initialized, Role: FOLLOWER
I20260619 09:04:52.582321 20671 consensus_queue.cc:260] T 00000000000000000000000000000000 P 37af3c32820d439fa711652c987de0ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37af3c32820d439fa711652c987de0ef" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33343 } }
I20260619 09:04:52.582407 20671 raft_consensus.cc:388] T 00000000000000000000000000000000 P 37af3c32820d439fa711652c987de0ef [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 09:04:52.582448 20671 raft_consensus.cc:482] T 00000000000000000000000000000000 P 37af3c32820d439fa711652c987de0ef [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 09:04:52.582496 20671 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 37af3c32820d439fa711652c987de0ef [term 0 FOLLOWER]: Advancing to term 1
Jun 19 09:04:52 dist-test-slave-mtdr krb5kdc[20590](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859892, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260619 09:04:52.583166 20671 raft_consensus.cc:504] T 00000000000000000000000000000000 P 37af3c32820d439fa711652c987de0ef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37af3c32820d439fa711652c987de0ef" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33343 } }
I20260619 09:04:52.583261 20671 leader_election.cc:302] T 00000000000000000000000000000000 P 37af3c32820d439fa711652c987de0ef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 37af3c32820d439fa711652c987de0ef; no voters:
I20260619 09:04:52.583439 20671 leader_election.cc:288] T 00000000000000000000000000000000 P 37af3c32820d439fa711652c987de0ef [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 09:04:52.583482 20676 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 37af3c32820d439fa711652c987de0ef [term 1 FOLLOWER]: Leader election won for term 1
I20260619 09:04:52.583616 20676 raft_consensus.cc:686] T 00000000000000000000000000000000 P 37af3c32820d439fa711652c987de0ef [term 1 LEADER]: Becoming Leader. State: Replica: 37af3c32820d439fa711652c987de0ef, State: Running, Role: LEADER
I20260619 09:04:52.583691 20671 sys_catalog.cc:564] T 00000000000000000000000000000000 P 37af3c32820d439fa711652c987de0ef [sys.catalog]: configured and running, proceeding with master startup.
I20260619 09:04:52.583707 20676 consensus_queue.cc:237] T 00000000000000000000000000000000 P 37af3c32820d439fa711652c987de0ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37af3c32820d439fa711652c987de0ef" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33343 } }
I20260619 09:04:52.584196 20677 sys_catalog.cc:455] T 00000000000000000000000000000000 P 37af3c32820d439fa711652c987de0ef [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "37af3c32820d439fa711652c987de0ef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37af3c32820d439fa711652c987de0ef" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33343 } } }
I20260619 09:04:52.584451 20677 sys_catalog.cc:458] T 00000000000000000000000000000000 P 37af3c32820d439fa711652c987de0ef [sys.catalog]: This master's current role is: LEADER
I20260619 09:04:52.584376 20678 sys_catalog.cc:455] T 00000000000000000000000000000000 P 37af3c32820d439fa711652c987de0ef [sys.catalog]: SysCatalogTable state changed. Reason: New leader 37af3c32820d439fa711652c987de0ef. Latest consensus state: current_term: 1 leader_uuid: "37af3c32820d439fa711652c987de0ef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37af3c32820d439fa711652c987de0ef" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33343 } } }
I20260619 09:04:52.584631 20678 sys_catalog.cc:458] T 00000000000000000000000000000000 P 37af3c32820d439fa711652c987de0ef [sys.catalog]: This master's current role is: LEADER
I20260619 09:04:52.584797 20686 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 09:04:52.585343 20686 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 09:04:52.587103 20686 catalog_manager.cc:1347] Generated new cluster ID: 330bba66fc7e4371acc11bc0d58c2847
I20260619 09:04:52.587157 20686 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 09:04:52.587802 20674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:04:52.577117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49993 (local address 127.0.0.1:33343)
0619 09:04:52.577280 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0619 09:04:52.577287 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 09:04:52.577311 (+ 24us) server_negotiation.cc:407] Connection header received
0619 09:04:52.577689 (+ 378us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:04:52.577695 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:04:52.577740 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:04:52.577850 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:04:52.578680 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:04:52.579672 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:04:52.580455 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:04:52.580861 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:04:52.580962 (+ 101us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:04:52.583418 (+ 2456us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:04:52.583423 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:04:52.583424 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:04:52.583437 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:04:52.585828 (+ 2391us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:04:52.586263 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:04:52.586266 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:04:52.586267 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:04:52.586366 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:04:52.586593 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:04:52.586593 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:04:52.586594 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:04:52.586898 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:04:52.587027 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0619 09:04:52.587505 (+ 478us) server_negotiation.cc:299] Negotiation successful
0619 09:04:52.587599 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":62,"threads_started":1}
I20260619 09:04:52.593513 20686 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 09:04:52.593914 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 09:04:52.594173 20686 catalog_manager.cc:1504] Loading token signing keys...
I20260619 09:04:52.602468 20674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:04:52.595371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37055 (local address 127.0.0.1:33343)
0619 09:04:52.595402 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0619 09:04:52.595404 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:04:52.595526 (+ 122us) server_negotiation.cc:407] Connection header received
0619 09:04:52.595562 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:04:52.595563 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:04:52.595574 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:04:52.595644 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:04:52.596214 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:04:52.596985 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:04:52.597977 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:04:52.598309 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:04:52.598386 (+ 77us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:04:52.599704 (+ 1318us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:04:52.599706 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:04:52.599706 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:04:52.599714 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:04:52.601287 (+ 1573us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:04:52.601680 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:04:52.601681 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:04:52.601682 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:04:52.601755 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:04:52.601955 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:04:52.601955 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:04:52.601956 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:04:52.602088 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:04:52.602212 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0619 09:04:52.602300 (+ 88us) server_negotiation.cc:299] Negotiation successful
0619 09:04:52.602366 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260619 09:04:52.612844 20686 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 37af3c32820d439fa711652c987de0ef: Generated new TSK 0
I20260619 09:04:52.613072 20686 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 09:04:52.622426 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ez5NB/build/release/bin/kudu with pid 20603
W20260619 09:04:52.626147 20588 connection.cc:535] client connection to 127.0.0.1:33343 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33343): Connection reset by peer (error 104)
2026-06-19T09:04:52Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1224 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781859658134510-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 09:04:52 dist-test-slave-mtdr krb5kdc[20702](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 09:04:52 dist-test-slave-mtdr krb5kdc[20702](info): set up 2 sockets
Jun 19 09:04:52 dist-test-slave-mtdr krb5kdc[20702](info): commencing operation
krb5kdc: starting...
W20260619 09:04:53.598568 392 mini_kdc.cc:121] Time spent starting KDC: real 0.959s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781859658134510-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781859658134510-392-0/krb5kdc/test-user.keytab.
Jun 19 09:04:53 dist-test-slave-mtdr krb5kdc[20702](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859893, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T09:04:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T09:04:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781859658134510-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781859658134510-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 09:04:53.709138 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ez5NB/build/release/bin/kudu
/tmp/dist-test-task4ez5NB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781859658134510-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781859658134510-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781859658134510-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781859658134510-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46355
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781859658134510-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46355
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781859658134510-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781859658134510-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781859658134510-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 09:04:53.786206 20715 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 09:04:53.786432 20715 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 09:04:53.786455 20715 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 09:04:53.787624 20715 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 09:04:53.787662 20715 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 09:04:53.787673 20715 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 09:04:53.787685 20715 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 09:04:53.787695 20715 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 09:04:53.789127 20715 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781859658134510-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781859658134510-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46355
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781859658134510-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46355
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781859658134510-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20715
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781859658134510-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 09:04:53.789325 20715 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 09:04:53.789497 20715 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 09:04:53.794178 20720 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:04:53.794173 20721 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:04:53.794256 20723 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:04:53.794440 20715 server_base.cc:1034] running on GCE node
I20260619 09:04:53.794763 20715 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 09:04:53.795055 20715 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 09:04:53.796207 20715 hybrid_clock.cc:648] HybridClock initialized: now 1781859893796189 us; error 34 us; skew 500 ppm
Jun 19 09:04:53 dist-test-slave-mtdr krb5kdc[20702](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859893, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 09:04:53.798702 20715 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 09:04:53.799228 20715 webserver.cc:460] Webserver started at http://127.0.0.1:45307/ using document root <none> and password file <none>
I20260619 09:04:53.799417 20715 fs_manager.cc:359] Metadata directory not provided
I20260619 09:04:53.799455 20715 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 09:04:53.799544 20715 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 09:04:53.800544 20715 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781859658134510-392-0/minicluster-data/master-0/data/instance:
uuid: "8d3684ef78c247c0af55b1d0aa33ffae"
format_stamp: "Formatted at 2026-06-19 09:04:53 on dist-test-slave-mtdr"
server_key: "87f08e7ed73bfd72a11ddf74def67412"
server_key_iv: "eabca1e403b67167b9001f111ff2f6b3"
server_key_version: "encryptionkey@0"
I20260619 09:04:53.800836 20715 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781859658134510-392-0/minicluster-data/master-0/wal/instance:
uuid: "8d3684ef78c247c0af55b1d0aa33ffae"
format_stamp: "Formatted at 2026-06-19 09:04:53 on dist-test-slave-mtdr"
server_key: "87f08e7ed73bfd72a11ddf74def67412"
server_key_iv: "eabca1e403b67167b9001f111ff2f6b3"
server_key_version: "encryptionkey@0"
I20260619 09:04:53.802070 20715 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 09:04:53.802868 20730 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 09:04:53.803010 20715 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260619 09:04:53.803095 20715 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781859658134510-392-0/minicluster-data/master-0/data,/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781859658134510-392-0/minicluster-data/master-0/wal
uuid: "8d3684ef78c247c0af55b1d0aa33ffae"
format_stamp: "Formatted at 2026-06-19 09:04:53 on dist-test-slave-mtdr"
server_key: "87f08e7ed73bfd72a11ddf74def67412"
server_key_iv: "eabca1e403b67167b9001f111ff2f6b3"
server_key_version: "encryptionkey@0"
I20260619 09:04:53.803161 20715 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781859658134510-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781859658134510-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781859658134510-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 09:04:53.816442 20715 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 09:04:53.818310 20715 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 09:04:53.818444 20715 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 09:04:53.822135 20715 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46355
I20260619 09:04:53.822170 20782 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46355 every 8 connection(s)
I20260619 09:04:53.822481 20715 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781859658134510-392-0/minicluster-data/master-0/data/info.pb
I20260619 09:04:53.823226 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ez5NB/build/release/bin/kudu as pid 20715
I20260619 09:04:53.823331 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781859658134510-392-0/minicluster-data/master-0/wal/instance
I20260619 09:04:53.823328 20783 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 09:04:53.823531 392 external_mini_cluster.cc:1428] Setting key addaa454fd11d7588b37f55ef4dc5e38
I20260619 09:04:53.826202 20783 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d3684ef78c247c0af55b1d0aa33ffae: Bootstrap starting.
I20260619 09:04:53.826893 20783 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8d3684ef78c247c0af55b1d0aa33ffae: Neither blocks nor log segments found. Creating new log.
I20260619 09:04:53.827170 20783 log.cc:824] T 00000000000000000000000000000000 P 8d3684ef78c247c0af55b1d0aa33ffae: Log is configured to *not* fsync() on all Append() calls
I20260619 09:04:53.827755 20783 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d3684ef78c247c0af55b1d0aa33ffae: No bootstrap required, opened a new log
I20260619 09:04:53.828675 20786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:04:53.823984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52535 (local address 127.0.0.1:46355)
0619 09:04:53.824141 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0619 09:04:53.824148 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 09:04:53.824172 (+ 24us) server_negotiation.cc:407] Connection header received
0619 09:04:53.824643 (+ 471us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:04:53.824649 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:04:53.824696 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:04:53.824818 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:04:53.825037 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:04:53.825893 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:04:53.826944 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:04:53.827372 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:04:53.827470 (+ 98us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:04:53.828101 (+ 631us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:04:53.828104 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:04:53.828105 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 09:04:53.828253 (+ 148us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:04:53.828282 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 09:04:53.828293 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:04:53.828355 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0619 09:04:53.828455 (+ 100us) server_negotiation.cc:299] Negotiation successful
0619 09:04:53.828476 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":61,"threads_started":1}
I20260619 09:04:53.829573 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 09:04:53.829576 20783 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8d3684ef78c247c0af55b1d0aa33ffae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d3684ef78c247c0af55b1d0aa33ffae" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46355 } }
I20260619 09:04:53.829715 20783 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8d3684ef78c247c0af55b1d0aa33ffae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 09:04:53.829754 20783 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8d3684ef78c247c0af55b1d0aa33ffae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d3684ef78c247c0af55b1d0aa33ffae, State: Initialized, Role: FOLLOWER
I20260619 09:04:53.829864 20783 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8d3684ef78c247c0af55b1d0aa33ffae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d3684ef78c247c0af55b1d0aa33ffae" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46355 } }
I20260619 09:04:53.829928 20783 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8d3684ef78c247c0af55b1d0aa33ffae [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 09:04:53.829952 20783 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8d3684ef78c247c0af55b1d0aa33ffae [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 09:04:53.829996 20783 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8d3684ef78c247c0af55b1d0aa33ffae [term 0 FOLLOWER]: Advancing to term 1
I20260619 09:04:53.830482 20783 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8d3684ef78c247c0af55b1d0aa33ffae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d3684ef78c247c0af55b1d0aa33ffae" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46355 } }
I20260619 09:04:53.830580 20783 leader_election.cc:302] T 00000000000000000000000000000000 P 8d3684ef78c247c0af55b1d0aa33ffae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8d3684ef78c247c0af55b1d0aa33ffae; no voters:
I20260619 09:04:53.830737 20783 leader_election.cc:288] T 00000000000000000000000000000000 P 8d3684ef78c247c0af55b1d0aa33ffae [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 09:04:53.830765 20792 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8d3684ef78c247c0af55b1d0aa33ffae [term 1 FOLLOWER]: Leader election won for term 1
I20260619 09:04:53.830948 20792 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8d3684ef78c247c0af55b1d0aa33ffae [term 1 LEADER]: Becoming Leader. State: Replica: 8d3684ef78c247c0af55b1d0aa33ffae, State: Running, Role: LEADER
I20260619 09:04:53.831041 20783 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8d3684ef78c247c0af55b1d0aa33ffae [sys.catalog]: configured and running, proceeding with master startup.
I20260619 09:04:53.831066 20792 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8d3684ef78c247c0af55b1d0aa33ffae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d3684ef78c247c0af55b1d0aa33ffae" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46355 } }
I20260619 09:04:53.831357 20792 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d3684ef78c247c0af55b1d0aa33ffae [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8d3684ef78c247c0af55b1d0aa33ffae. Latest consensus state: current_term: 1 leader_uuid: "8d3684ef78c247c0af55b1d0aa33ffae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d3684ef78c247c0af55b1d0aa33ffae" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46355 } } }
I20260619 09:04:53.831382 20794 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d3684ef78c247c0af55b1d0aa33ffae [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8d3684ef78c247c0af55b1d0aa33ffae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d3684ef78c247c0af55b1d0aa33ffae" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46355 } } }
I20260619 09:04:53.831490 20792 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d3684ef78c247c0af55b1d0aa33ffae [sys.catalog]: This master's current role is: LEADER
I20260619 09:04:53.831535 20794 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d3684ef78c247c0af55b1d0aa33ffae [sys.catalog]: This master's current role is: LEADER
W20260619 09:04:53.832720 20809 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8d3684ef78c247c0af55b1d0aa33ffae: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 09:04:53.832877 20809 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 09:04:53.832974 20801 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 09:04:53.833177 20801 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 09:04:53.834578 20801 catalog_manager.cc:1347] Generated new cluster ID: b66be7b287f74eff90b598b4d2eb2c52
I20260619 09:04:53.834630 20801 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 09:04:53.835003 20786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:04:53.830873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37923 (local address 127.0.0.1:46355)
0619 09:04:53.830927 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0619 09:04:53.830929 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:04:53.831881 (+ 952us) server_negotiation.cc:407] Connection header received
0619 09:04:53.831957 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:04:53.831958 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:04:53.831967 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:04:53.832014 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:04:53.832170 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:04:53.832727 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:04:53.833680 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:04:53.833965 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:04:53.834026 (+ 61us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:04:53.834733 (+ 707us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:04:53.834734 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:04:53.834734 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 09:04:53.834738 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:04:53.834753 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 09:04:53.834758 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:04:53.834839 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0619 09:04:53.834921 (+ 82us) server_negotiation.cc:299] Negotiation successful
0619 09:04:53.834932 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260619 09:04:53.860248 20801 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 09:04:53.861023 20801 catalog_manager.cc:1504] Loading token signing keys...
I20260619 09:04:53.865417 20801 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8d3684ef78c247c0af55b1d0aa33ffae: Generated new TSK 0
I20260619 09:04:53.865597 20801 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 09:04:53.871924 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ez5NB/build/release/bin/kudu with pid 20715
W20260619 09:04:53.875768 20700 connection.cc:535] client connection to 127.0.0.1:46355 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46355): Connection reset by peer (error 104)
2026-06-19T09:04:53Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1249 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781859658134510-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 09:04:53 dist-test-slave-mtdr krb5kdc[20814](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 09:04:53 dist-test-slave-mtdr krb5kdc[20814](info): set up 2 sockets
Jun 19 09:04:53 dist-test-slave-mtdr krb5kdc[20814](info): commencing operation
krb5kdc: starting...
W20260619 09:04:54.815131 392 mini_kdc.cc:121] Time spent starting KDC: real 0.926s user 0.004s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781859658134510-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781859658134510-392-0/krb5kdc/test-user.keytab.
Jun 19 09:04:54 dist-test-slave-mtdr krb5kdc[20814](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859894, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T09:04:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T09:04:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781859658134510-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781859658134510-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 09:04:54.929591 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ez5NB/build/release/bin/kudu
/tmp/dist-test-task4ez5NB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781859658134510-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781859658134510-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781859658134510-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781859658134510-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36107
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781859658134510-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41659
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36107
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781859658134510-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781859658134510-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781859658134510-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 09:04:55.007314 20827 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 09:04:55.007506 20827 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 09:04:55.007527 20827 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 09:04:55.008652 20827 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 09:04:55.008693 20827 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 09:04:55.008706 20827 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 09:04:55.008718 20827 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 09:04:55.008728 20827 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 09:04:55.010170 20827 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41659
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781859658134510-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781859658134510-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36107
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781859658134510-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36107
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781859658134510-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20827
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781859658134510-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 09:04:55.010390 20827 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 09:04:55.010586 20827 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 09:04:55.015123 20832 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:04:55.015105 20833 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:04:55.015180 20827 server_base.cc:1034] running on GCE node
W20260619 09:04:55.015105 20835 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:04:55.015558 20827 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 09:04:55.015834 20827 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 09:04:55.016999 20827 hybrid_clock.cc:648] HybridClock initialized: now 1781859895016969 us; error 42 us; skew 500 ppm
Jun 19 09:04:55 dist-test-slave-mtdr krb5kdc[20814](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859895, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 09:04:55.019555 20827 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 09:04:55.019986 20827 webserver.cc:460] Webserver started at http://127.0.0.1:40413/ using document root <none> and password file <none>
I20260619 09:04:55.020197 20827 fs_manager.cc:359] Metadata directory not provided
I20260619 09:04:55.020244 20827 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 09:04:55.020361 20827 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 09:04:55.021405 20827 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781859658134510-392-0/minicluster-data/master-0/data/instance:
uuid: "a791abb1117b455cb467718d1ed58958"
format_stamp: "Formatted at 2026-06-19 09:04:55 on dist-test-slave-mtdr"
server_key: "64cbc3f5f36d4ca1dcb8748464524b6a"
server_key_iv: "7bc2ba30f26ebc33447a5d06dec68833"
server_key_version: "encryptionkey@0"
I20260619 09:04:55.021724 20827 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781859658134510-392-0/minicluster-data/master-0/wal/instance:
uuid: "a791abb1117b455cb467718d1ed58958"
format_stamp: "Formatted at 2026-06-19 09:04:55 on dist-test-slave-mtdr"
server_key: "64cbc3f5f36d4ca1dcb8748464524b6a"
server_key_iv: "7bc2ba30f26ebc33447a5d06dec68833"
server_key_version: "encryptionkey@0"
I20260619 09:04:55.023031 20827 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260619 09:04:55.023757 20842 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 09:04:55.023916 20827 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 09:04:55.024000 20827 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781859658134510-392-0/minicluster-data/master-0/data,/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781859658134510-392-0/minicluster-data/master-0/wal
uuid: "a791abb1117b455cb467718d1ed58958"
format_stamp: "Formatted at 2026-06-19 09:04:55 on dist-test-slave-mtdr"
server_key: "64cbc3f5f36d4ca1dcb8748464524b6a"
server_key_iv: "7bc2ba30f26ebc33447a5d06dec68833"
server_key_version: "encryptionkey@0"
I20260619 09:04:55.024144 20827 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781859658134510-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781859658134510-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781859658134510-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 09:04:55.027408 20827 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 09:04:55.029413 20827 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 09:04:55.029536 20827 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 09:04:55.033098 20827 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36107
I20260619 09:04:55.033135 20894 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36107 every 8 connection(s)
I20260619 09:04:55.033420 20827 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781859658134510-392-0/minicluster-data/master-0/data/info.pb
I20260619 09:04:55.033988 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ez5NB/build/release/bin/kudu as pid 20827
I20260619 09:04:55.034085 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781859658134510-392-0/minicluster-data/master-0/wal/instance
I20260619 09:04:55.034256 392 external_mini_cluster.cc:1428] Setting key 4ee1e9dfd947668bf6925eae4e786140
I20260619 09:04:55.034399 20895 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 09:04:55.036638 20898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:04:55.034696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41055 (local address 127.0.0.1:36107)
0619 09:04:55.034874 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0619 09:04:55.034882 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 09:04:55.034906 (+ 24us) server_negotiation.cc:407] Connection header received
0619 09:04:55.035414 (+ 508us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:04:55.035420 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:04:55.035445 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:04:55.035558 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:04:55.036206 (+ 648us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:04:55.036209 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:04:55.036209 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 09:04:55.036213 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:04:55.036249 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 09:04:55.036259 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:04:55.036331 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0619 09:04:55.036405 (+ 74us) server_negotiation.cc:299] Negotiation successful
0619 09:04:55.036444 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":69,"threads_started":1}
I20260619 09:04:55.037654 20895 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a791abb1117b455cb467718d1ed58958: Bootstrap starting.
I20260619 09:04:55.037761 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 09:04:55.038239 20895 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a791abb1117b455cb467718d1ed58958: Neither blocks nor log segments found. Creating new log.
I20260619 09:04:55.038493 20895 log.cc:824] T 00000000000000000000000000000000 P a791abb1117b455cb467718d1ed58958: Log is configured to *not* fsync() on all Append() calls
I20260619 09:04:55.039219 20895 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a791abb1117b455cb467718d1ed58958: No bootstrap required, opened a new log
I20260619 09:04:55.039994 20898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:04:55.039077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48201 (local address 127.0.0.1:36107)
0619 09:04:55.039114 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0619 09:04:55.039116 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:04:55.039234 (+ 118us) server_negotiation.cc:407] Connection header received
0619 09:04:55.039298 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:04:55.039299 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:04:55.039304 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:04:55.039359 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:04:55.039822 (+ 463us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:04:55.039823 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:04:55.039823 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 09:04:55.039826 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:04:55.039837 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 09:04:55.039841 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:04:55.039880 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0619 09:04:55.039930 (+ 50us) server_negotiation.cc:299] Negotiation successful
0619 09:04:55.039939 (+ 9us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260619 09:04:55.041246 20895 raft_consensus.cc:348] T 00000000000000000000000000000000 P a791abb1117b455cb467718d1ed58958 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a791abb1117b455cb467718d1ed58958" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36107 } }
I20260619 09:04:55.041378 20895 raft_consensus.cc:374] T 00000000000000000000000000000000 P a791abb1117b455cb467718d1ed58958 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 09:04:55.041399 20895 raft_consensus.cc:729] T 00000000000000000000000000000000 P a791abb1117b455cb467718d1ed58958 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a791abb1117b455cb467718d1ed58958, State: Initialized, Role: FOLLOWER
I20260619 09:04:55.041493 20895 consensus_queue.cc:260] T 00000000000000000000000000000000 P a791abb1117b455cb467718d1ed58958 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a791abb1117b455cb467718d1ed58958" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36107 } }
I20260619 09:04:55.041576 20895 raft_consensus.cc:388] T 00000000000000000000000000000000 P a791abb1117b455cb467718d1ed58958 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 09:04:55.041599 20895 raft_consensus.cc:482] T 00000000000000000000000000000000 P a791abb1117b455cb467718d1ed58958 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 09:04:55.041625 20895 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a791abb1117b455cb467718d1ed58958 [term 0 FOLLOWER]: Advancing to term 1
I20260619 09:04:55.042085 20895 raft_consensus.cc:504] T 00000000000000000000000000000000 P a791abb1117b455cb467718d1ed58958 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a791abb1117b455cb467718d1ed58958" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36107 } }
I20260619 09:04:55.042176 20895 leader_election.cc:302] T 00000000000000000000000000000000 P a791abb1117b455cb467718d1ed58958 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a791abb1117b455cb467718d1ed58958; no voters:
I20260619 09:04:55.042300 20895 leader_election.cc:288] T 00000000000000000000000000000000 P a791abb1117b455cb467718d1ed58958 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 09:04:55.042352 20905 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a791abb1117b455cb467718d1ed58958 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 09:04:55.042501 20895 sys_catalog.cc:564] T 00000000000000000000000000000000 P a791abb1117b455cb467718d1ed58958 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 09:04:55.042516 20905 raft_consensus.cc:686] T 00000000000000000000000000000000 P a791abb1117b455cb467718d1ed58958 [term 1 LEADER]: Becoming Leader. State: Replica: a791abb1117b455cb467718d1ed58958, State: Running, Role: LEADER
I20260619 09:04:55.042688 20905 consensus_queue.cc:237] T 00000000000000000000000000000000 P a791abb1117b455cb467718d1ed58958 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a791abb1117b455cb467718d1ed58958" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36107 } }
I20260619 09:04:55.043138 20906 sys_catalog.cc:455] T 00000000000000000000000000000000 P a791abb1117b455cb467718d1ed58958 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a791abb1117b455cb467718d1ed58958" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a791abb1117b455cb467718d1ed58958" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36107 } } }
I20260619 09:04:55.043350 20906 sys_catalog.cc:458] T 00000000000000000000000000000000 P a791abb1117b455cb467718d1ed58958 [sys.catalog]: This master's current role is: LEADER
I20260619 09:04:55.043085 20907 sys_catalog.cc:455] T 00000000000000000000000000000000 P a791abb1117b455cb467718d1ed58958 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a791abb1117b455cb467718d1ed58958. Latest consensus state: current_term: 1 leader_uuid: "a791abb1117b455cb467718d1ed58958" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a791abb1117b455cb467718d1ed58958" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36107 } } }
I20260619 09:04:55.043581 20907 sys_catalog.cc:458] T 00000000000000000000000000000000 P a791abb1117b455cb467718d1ed58958 [sys.catalog]: This master's current role is: LEADER
I20260619 09:04:55.043812 20915 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 09:04:55.044190 20915 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 09:04:55.045506 20915 catalog_manager.cc:1347] Generated new cluster ID: 49cc1315ab4a4241a5770cf3bf365b3d
I20260619 09:04:55.045554 20915 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 09:04:55.061676 20915 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 09:04:55.061854 20915 catalog_manager.cc:1504] Loading token signing keys...
I20260619 09:04:55.074347 20915 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a791abb1117b455cb467718d1ed58958: Generated new TSK 0
I20260619 09:04:55.074543 20915 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 09:04:55.080890 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ez5NB/build/release/bin/kudu with pid 20827
2026-06-19T09:04:55Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1208 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781859658134510-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 09:04:55 dist-test-slave-mtdr krb5kdc[20926](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 09:04:55 dist-test-slave-mtdr krb5kdc[20926](info): set up 2 sockets
Jun 19 09:04:55 dist-test-slave-mtdr krb5kdc[20926](info): commencing operation
krb5kdc: starting...
W20260619 09:04:56.042552 392 mini_kdc.cc:121] Time spent starting KDC: real 0.945s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781859658134510-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781859658134510-392-0/krb5kdc/test-user.keytab.
Jun 19 09:04:56 dist-test-slave-mtdr krb5kdc[20926](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859896, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T09:04:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T09:04:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781859658134510-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781859658134510-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 09:04:56.153117 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ez5NB/build/release/bin/kudu
/tmp/dist-test-task4ez5NB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781859658134510-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781859658134510-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781859658134510-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781859658134510-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46545
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781859658134510-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43355
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46545
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781859658134510-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781859658134510-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781859658134510-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 09:04:56.230448 20939 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 09:04:56.230652 20939 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 09:04:56.230681 20939 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 09:04:56.231846 20939 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 09:04:56.231889 20939 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 09:04:56.231907 20939 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 09:04:56.231923 20939 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 09:04:56.231937 20939 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 09:04:56.233358 20939 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43355
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781859658134510-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781859658134510-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46545
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781859658134510-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46545
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781859658134510-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20939
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781859658134510-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 09:04:56.233592 20939 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 09:04:56.233791 20939 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 09:04:56.238349 20945 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:04:56.238406 20944 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:04:56.238364 20947 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:04:56.238511 20939 server_base.cc:1034] running on GCE node
I20260619 09:04:56.238754 20939 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 09:04:56.239027 20939 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 09:04:56.240202 20939 hybrid_clock.cc:648] HybridClock initialized: now 1781859896240186 us; error 31 us; skew 500 ppm
Jun 19 09:04:56 dist-test-slave-mtdr krb5kdc[20926](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859896, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 09:04:56.242514 20939 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 09:04:56.242961 20939 webserver.cc:460] Webserver started at http://127.0.0.1:43603/ using document root <none> and password file <none>
I20260619 09:04:56.243165 20939 fs_manager.cc:359] Metadata directory not provided
I20260619 09:04:56.243216 20939 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 09:04:56.243336 20939 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 09:04:56.244349 20939 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781859658134510-392-0/minicluster-data/master-0/data/instance:
uuid: "5c87cf6d1dd74929804c00eb65509fa2"
format_stamp: "Formatted at 2026-06-19 09:04:56 on dist-test-slave-mtdr"
server_key: "8d7a54df296277017244d814f32821a7"
server_key_iv: "35258bd6c37bca812129c16aff78b3ee"
server_key_version: "encryptionkey@0"
I20260619 09:04:56.244670 20939 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781859658134510-392-0/minicluster-data/master-0/wal/instance:
uuid: "5c87cf6d1dd74929804c00eb65509fa2"
format_stamp: "Formatted at 2026-06-19 09:04:56 on dist-test-slave-mtdr"
server_key: "8d7a54df296277017244d814f32821a7"
server_key_iv: "35258bd6c37bca812129c16aff78b3ee"
server_key_version: "encryptionkey@0"
I20260619 09:04:56.245893 20939 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 09:04:56.246683 20954 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 09:04:56.246886 20939 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 09:04:56.246966 20939 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781859658134510-392-0/minicluster-data/master-0/data,/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781859658134510-392-0/minicluster-data/master-0/wal
uuid: "5c87cf6d1dd74929804c00eb65509fa2"
format_stamp: "Formatted at 2026-06-19 09:04:56 on dist-test-slave-mtdr"
server_key: "8d7a54df296277017244d814f32821a7"
server_key_iv: "35258bd6c37bca812129c16aff78b3ee"
server_key_version: "encryptionkey@0"
I20260619 09:04:56.247018 20939 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781859658134510-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781859658134510-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781859658134510-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 09:04:56.264813 20939 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 09:04:56.266624 20939 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 09:04:56.266727 20939 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 09:04:56.270364 20939 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46545
I20260619 09:04:56.270423 21006 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46545 every 8 connection(s)
I20260619 09:04:56.270709 20939 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781859658134510-392-0/minicluster-data/master-0/data/info.pb
I20260619 09:04:56.271512 21007 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 09:04:56.273712 21007 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c87cf6d1dd74929804c00eb65509fa2: Bootstrap starting.
I20260619 09:04:56.274251 21007 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5c87cf6d1dd74929804c00eb65509fa2: Neither blocks nor log segments found. Creating new log.
I20260619 09:04:56.274494 21007 log.cc:824] T 00000000000000000000000000000000 P 5c87cf6d1dd74929804c00eb65509fa2: Log is configured to *not* fsync() on all Append() calls
I20260619 09:04:56.275137 21007 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c87cf6d1dd74929804c00eb65509fa2: No bootstrap required, opened a new log
I20260619 09:04:56.276641 21007 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5c87cf6d1dd74929804c00eb65509fa2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c87cf6d1dd74929804c00eb65509fa2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46545 } }
I20260619 09:04:56.276774 21007 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5c87cf6d1dd74929804c00eb65509fa2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 09:04:56.276804 21007 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5c87cf6d1dd74929804c00eb65509fa2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c87cf6d1dd74929804c00eb65509fa2, State: Initialized, Role: FOLLOWER
I20260619 09:04:56.276947 21007 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5c87cf6d1dd74929804c00eb65509fa2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c87cf6d1dd74929804c00eb65509fa2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46545 } }
I20260619 09:04:56.277022 21007 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5c87cf6d1dd74929804c00eb65509fa2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 09:04:56.277053 21007 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5c87cf6d1dd74929804c00eb65509fa2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 09:04:56.277100 21007 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5c87cf6d1dd74929804c00eb65509fa2 [term 0 FOLLOWER]: Advancing to term 1
I20260619 09:04:56.277335 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ez5NB/build/release/bin/kudu as pid 20939
I20260619 09:04:56.277429 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781859658134510-392-0/minicluster-data/master-0/wal/instance
I20260619 09:04:56.277599 392 external_mini_cluster.cc:1428] Setting key a7507ef503485d2b586ef23ed9020b8d
I20260619 09:04:56.277644 21007 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5c87cf6d1dd74929804c00eb65509fa2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c87cf6d1dd74929804c00eb65509fa2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46545 } }
I20260619 09:04:56.277747 21007 leader_election.cc:302] T 00000000000000000000000000000000 P 5c87cf6d1dd74929804c00eb65509fa2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5c87cf6d1dd74929804c00eb65509fa2; no voters:
I20260619 09:04:56.277906 21007 leader_election.cc:288] T 00000000000000000000000000000000 P 5c87cf6d1dd74929804c00eb65509fa2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 09:04:56.277961 21010 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5c87cf6d1dd74929804c00eb65509fa2 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 09:04:56.278115 21010 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5c87cf6d1dd74929804c00eb65509fa2 [term 1 LEADER]: Becoming Leader. State: Replica: 5c87cf6d1dd74929804c00eb65509fa2, State: Running, Role: LEADER
I20260619 09:04:56.278211 21010 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5c87cf6d1dd74929804c00eb65509fa2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c87cf6d1dd74929804c00eb65509fa2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46545 } }
I20260619 09:04:56.278244 21007 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5c87cf6d1dd74929804c00eb65509fa2 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 09:04:56.278549 21010 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c87cf6d1dd74929804c00eb65509fa2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5c87cf6d1dd74929804c00eb65509fa2. Latest consensus state: current_term: 1 leader_uuid: "5c87cf6d1dd74929804c00eb65509fa2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c87cf6d1dd74929804c00eb65509fa2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46545 } } }
I20260619 09:04:56.278625 21010 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c87cf6d1dd74929804c00eb65509fa2 [sys.catalog]: This master's current role is: LEADER
I20260619 09:04:56.278570 21014 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c87cf6d1dd74929804c00eb65509fa2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5c87cf6d1dd74929804c00eb65509fa2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c87cf6d1dd74929804c00eb65509fa2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46545 } } }
I20260619 09:04:56.278765 21014 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c87cf6d1dd74929804c00eb65509fa2 [sys.catalog]: This master's current role is: LEADER
I20260619 09:04:56.279055 21016 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 09:04:56.279613 21016 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 09:04:56.281219 21016 catalog_manager.cc:1347] Generated new cluster ID: 0ccc5898c6d74beb8092f90959652154
I20260619 09:04:56.281265 21016 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 19 09:04:56 dist-test-slave-mtdr krb5kdc[20926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859896, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260619 09:04:56.289012 21016 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 09:04:56.289327 21013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:04:56.278002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35395 (local address 127.0.0.1:46545)
0619 09:04:56.278161 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0619 09:04:56.278170 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 09:04:56.278918 (+ 748us) server_negotiation.cc:407] Connection header received
0619 09:04:56.279082 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:04:56.279089 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:04:56.279151 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:04:56.280051 (+ 900us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:04:56.280199 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:04:56.281430 (+ 1231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:04:56.282476 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:04:56.283085 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:04:56.283179 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:04:56.285805 (+ 2626us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:04:56.285809 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:04:56.285810 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:04:56.285822 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:04:56.287687 (+ 1865us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:04:56.288071 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:04:56.288075 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:04:56.288075 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:04:56.288207 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:04:56.288400 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:04:56.288401 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:04:56.288401 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:04:56.288693 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:04:56.288844 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0619 09:04:56.288978 (+ 134us) server_negotiation.cc:299] Negotiation successful
0619 09:04:56.289117 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":64,"threads_started":1}
I20260619 09:04:56.289743 21016 catalog_manager.cc:1504] Loading token signing keys...
I20260619 09:04:56.290132 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 09:04:56.300652 21013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:04:56.295083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52417 (local address 127.0.0.1:46545)
0619 09:04:56.295116 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0619 09:04:56.295118 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:04:56.295197 (+ 79us) server_negotiation.cc:407] Connection header received
0619 09:04:56.295248 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:04:56.295249 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:04:56.295259 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:04:56.295303 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:04:56.295828 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:04:56.296345 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:04:56.297055 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:04:56.297305 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:04:56.297367 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:04:56.298446 (+ 1079us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:04:56.298447 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:04:56.298447 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:04:56.298453 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:04:56.299707 (+ 1254us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:04:56.299996 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:04:56.299997 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:04:56.299997 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:04:56.300048 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:04:56.300199 (+ 151us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:04:56.300200 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:04:56.300200 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:04:56.300301 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:04:56.300388 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0619 09:04:56.300490 (+ 102us) server_negotiation.cc:299] Negotiation successful
0619 09:04:56.300564 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260619 09:04:56.302506 21016 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5c87cf6d1dd74929804c00eb65509fa2: Generated new TSK 0
I20260619 09:04:56.302681 21016 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 09:04:56.304927 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ez5NB/build/release/bin/kudu with pid 20939
2026-06-19T09:04:56Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1224 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781859658134510-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 09:04:56 dist-test-slave-mtdr krb5kdc[21038](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 09:04:56 dist-test-slave-mtdr krb5kdc[21038](info): set up 2 sockets
Jun 19 09:04:56 dist-test-slave-mtdr krb5kdc[21038](info): commencing operation
krb5kdc: starting...
W20260619 09:04:57.258565 392 mini_kdc.cc:121] Time spent starting KDC: real 0.936s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781859658134510-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781859658134510-392-0/krb5kdc/test-user.keytab.
Jun 19 09:04:57 dist-test-slave-mtdr krb5kdc[21038](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859897, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T09:04:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T09:04:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781859658134510-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781859658134510-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 09:04:57.368847 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ez5NB/build/release/bin/kudu
/tmp/dist-test-task4ez5NB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781859658134510-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781859658134510-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781859658134510-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781859658134510-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35289
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781859658134510-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36665
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35289
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781859658134510-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781859658134510-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781859658134510-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 09:04:57.445740 21051 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 09:04:57.445940 21051 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 09:04:57.445961 21051 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 09:04:57.447108 21051 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 09:04:57.447149 21051 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 09:04:57.447160 21051 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 09:04:57.447171 21051 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 09:04:57.447181 21051 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 09:04:57.448624 21051 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36665
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781859658134510-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781859658134510-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35289
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781859658134510-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35289
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781859658134510-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21051
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781859658134510-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 09:04:57.448843 21051 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 09:04:57.449059 21051 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 09:04:57.453626 21056 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:04:57.453629 21057 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:04:57.453716 21051 server_base.cc:1034] running on GCE node
W20260619 09:04:57.453629 21059 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:04:57.454049 21051 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 09:04:57.454284 21051 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 09:04:57.455408 21051 hybrid_clock.cc:648] HybridClock initialized: now 1781859897455391 us; error 36 us; skew 500 ppm
Jun 19 09:04:57 dist-test-slave-mtdr krb5kdc[21038](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859897, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 09:04:57.457680 21051 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 09:04:57.458101 21051 webserver.cc:460] Webserver started at http://127.0.0.1:45019/ using document root <none> and password file <none>
I20260619 09:04:57.458298 21051 fs_manager.cc:359] Metadata directory not provided
I20260619 09:04:57.458348 21051 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 09:04:57.458452 21051 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 09:04:57.459496 21051 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781859658134510-392-0/minicluster-data/master-0/data/instance:
uuid: "605787acae91457eabf426158ba70b60"
format_stamp: "Formatted at 2026-06-19 09:04:57 on dist-test-slave-mtdr"
server_key: "e3b41eda84121ea71b6793f669db38ec"
server_key_iv: "8e69e2c7777edefda131d7bc873f2027"
server_key_version: "encryptionkey@0"
I20260619 09:04:57.459802 21051 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781859658134510-392-0/minicluster-data/master-0/wal/instance:
uuid: "605787acae91457eabf426158ba70b60"
format_stamp: "Formatted at 2026-06-19 09:04:57 on dist-test-slave-mtdr"
server_key: "e3b41eda84121ea71b6793f669db38ec"
server_key_iv: "8e69e2c7777edefda131d7bc873f2027"
server_key_version: "encryptionkey@0"
I20260619 09:04:57.461002 21051 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 09:04:57.461721 21066 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 09:04:57.461959 21051 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 09:04:57.462064 21051 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781859658134510-392-0/minicluster-data/master-0/data,/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781859658134510-392-0/minicluster-data/master-0/wal
uuid: "605787acae91457eabf426158ba70b60"
format_stamp: "Formatted at 2026-06-19 09:04:57 on dist-test-slave-mtdr"
server_key: "e3b41eda84121ea71b6793f669db38ec"
server_key_iv: "8e69e2c7777edefda131d7bc873f2027"
server_key_version: "encryptionkey@0"
I20260619 09:04:57.462168 21051 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781859658134510-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781859658134510-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781859658134510-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 09:04:57.474953 21051 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 09:04:57.477077 21051 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 09:04:57.477187 21051 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 09:04:57.481170 21051 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35289
I20260619 09:04:57.481231 21118 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35289 every 8 connection(s)
I20260619 09:04:57.481566 21051 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781859658134510-392-0/minicluster-data/master-0/data/info.pb
I20260619 09:04:57.482425 21119 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 09:04:57.482949 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ez5NB/build/release/bin/kudu as pid 21051
I20260619 09:04:57.483048 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781859658134510-392-0/minicluster-data/master-0/wal/instance
I20260619 09:04:57.483228 392 external_mini_cluster.cc:1428] Setting key c99e34f0ae38348d314db9dc43f112c6
I20260619 09:04:57.485116 21119 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 605787acae91457eabf426158ba70b60: Bootstrap starting.
I20260619 09:04:57.485687 21119 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 605787acae91457eabf426158ba70b60: Neither blocks nor log segments found. Creating new log.
I20260619 09:04:57.485948 21119 log.cc:824] T 00000000000000000000000000000000 P 605787acae91457eabf426158ba70b60: Log is configured to *not* fsync() on all Append() calls
I20260619 09:04:57.486564 21119 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 605787acae91457eabf426158ba70b60: No bootstrap required, opened a new log
I20260619 09:04:57.488291 21122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:04:57.483696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53425 (local address 127.0.0.1:35289)
0619 09:04:57.483848 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0619 09:04:57.483856 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 09:04:57.483881 (+ 25us) server_negotiation.cc:407] Connection header received
0619 09:04:57.484301 (+ 420us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:04:57.484308 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:04:57.484355 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:04:57.484489 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:04:57.484802 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:04:57.485715 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:04:57.486510 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:04:57.487000 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:04:57.487095 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:04:57.487736 (+ 641us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:04:57.487740 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:04:57.487741 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 09:04:57.487897 (+ 156us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:04:57.487925 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 09:04:57.487936 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:04:57.487988 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0619 09:04:57.488059 (+ 71us) server_negotiation.cc:299] Negotiation successful
0619 09:04:57.488119 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":62,"threads_started":1}
I20260619 09:04:57.488628 21119 raft_consensus.cc:348] T 00000000000000000000000000000000 P 605787acae91457eabf426158ba70b60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "605787acae91457eabf426158ba70b60" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35289 } }
I20260619 09:04:57.488771 21119 raft_consensus.cc:374] T 00000000000000000000000000000000 P 605787acae91457eabf426158ba70b60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 09:04:57.488804 21119 raft_consensus.cc:729] T 00000000000000000000000000000000 P 605787acae91457eabf426158ba70b60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 605787acae91457eabf426158ba70b60, State: Initialized, Role: FOLLOWER
I20260619 09:04:57.488921 21119 consensus_queue.cc:260] T 00000000000000000000000000000000 P 605787acae91457eabf426158ba70b60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "605787acae91457eabf426158ba70b60" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35289 } }
I20260619 09:04:57.489005 21119 raft_consensus.cc:388] T 00000000000000000000000000000000 P 605787acae91457eabf426158ba70b60 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 09:04:57.489048 21119 raft_consensus.cc:482] T 00000000000000000000000000000000 P 605787acae91457eabf426158ba70b60 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 09:04:57.489099 21119 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 605787acae91457eabf426158ba70b60 [term 0 FOLLOWER]: Advancing to term 1
I20260619 09:04:57.489130 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 09:04:57.489609 21119 raft_consensus.cc:504] T 00000000000000000000000000000000 P 605787acae91457eabf426158ba70b60 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "605787acae91457eabf426158ba70b60" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35289 } }
I20260619 09:04:57.489703 21119 leader_election.cc:302] T 00000000000000000000000000000000 P 605787acae91457eabf426158ba70b60 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 605787acae91457eabf426158ba70b60; no voters:
I20260619 09:04:57.489857 21119 leader_election.cc:288] T 00000000000000000000000000000000 P 605787acae91457eabf426158ba70b60 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 09:04:57.489907 21124 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 605787acae91457eabf426158ba70b60 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 09:04:57.490078 21124 raft_consensus.cc:686] T 00000000000000000000000000000000 P 605787acae91457eabf426158ba70b60 [term 1 LEADER]: Becoming Leader. State: Replica: 605787acae91457eabf426158ba70b60, State: Running, Role: LEADER
I20260619 09:04:57.490096 21119 sys_catalog.cc:564] T 00000000000000000000000000000000 P 605787acae91457eabf426158ba70b60 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 09:04:57.491374 21124 consensus_queue.cc:237] T 00000000000000000000000000000000 P 605787acae91457eabf426158ba70b60 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "605787acae91457eabf426158ba70b60" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35289 } }
I20260619 09:04:57.491889 21126 sys_catalog.cc:455] T 00000000000000000000000000000000 P 605787acae91457eabf426158ba70b60 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 605787acae91457eabf426158ba70b60. Latest consensus state: current_term: 1 leader_uuid: "605787acae91457eabf426158ba70b60" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "605787acae91457eabf426158ba70b60" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35289 } } }
I20260619 09:04:57.491981 21128 sys_catalog.cc:455] T 00000000000000000000000000000000 P 605787acae91457eabf426158ba70b60 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "605787acae91457eabf426158ba70b60" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "605787acae91457eabf426158ba70b60" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35289 } } }
I20260619 09:04:57.492238 21126 sys_catalog.cc:458] T 00000000000000000000000000000000 P 605787acae91457eabf426158ba70b60 [sys.catalog]: This master's current role is: LEADER
I20260619 09:04:57.492250 21128 sys_catalog.cc:458] T 00000000000000000000000000000000 P 605787acae91457eabf426158ba70b60 [sys.catalog]: This master's current role is: LEADER
W20260619 09:04:57.492367 21143 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 605787acae91457eabf426158ba70b60: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 09:04:57.492403 21143 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 09:04:57.492467 21145 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 09:04:57.492688 21145 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 09:04:57.493429 21122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:04:57.490485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33901 (local address 127.0.0.1:35289)
0619 09:04:57.490517 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0619 09:04:57.490519 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:04:57.490616 (+ 97us) server_negotiation.cc:407] Connection header received
0619 09:04:57.490670 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:04:57.490671 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:04:57.490679 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:04:57.490724 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:04:57.490891 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:04:57.491399 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:04:57.492212 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:04:57.492502 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:04:57.492544 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:04:57.493206 (+ 662us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:04:57.493207 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:04:57.493207 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 09:04:57.493211 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:04:57.493224 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 09:04:57.493228 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:04:57.493252 (+ 24us) server_negotiation.cc:1035] Waiting for connection context
0619 09:04:57.493344 (+ 92us) server_negotiation.cc:299] Negotiation successful
0619 09:04:57.493364 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260619 09:04:57.494239 21145 catalog_manager.cc:1347] Generated new cluster ID: 61a0f87832a84b0c84476f908224bee6
I20260619 09:04:57.494283 21145 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 09:04:57.502455 21145 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 09:04:57.503192 21145 catalog_manager.cc:1504] Loading token signing keys...
I20260619 09:04:57.513717 21145 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 605787acae91457eabf426158ba70b60: Generated new TSK 0
I20260619 09:04:57.513906 21145 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 09:04:57.515753 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ez5NB/build/release/bin/kudu with pid 21051
2026-06-19T09:04:57Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1210 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781859658134510-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 09:04:57 dist-test-slave-mtdr krb5kdc[21150](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 09:04:57 dist-test-slave-mtdr krb5kdc[21150](info): set up 2 sockets
Jun 19 09:04:57 dist-test-slave-mtdr krb5kdc[21150](info): commencing operation
krb5kdc: starting...
W20260619 09:04:58.470587 392 mini_kdc.cc:121] Time spent starting KDC: real 0.938s user 0.001s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781859658134510-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781859658134510-392-0/krb5kdc/test-user.keytab.
Jun 19 09:04:58 dist-test-slave-mtdr krb5kdc[21150](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859898, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T09:04:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T09:04:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781859658134510-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781859658134510-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 09:04:58.578194 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ez5NB/build/release/bin/kudu
/tmp/dist-test-task4ez5NB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781859658134510-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781859658134510-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781859658134510-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781859658134510-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40481
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781859658134510-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41433
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40481
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781859658134510-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781859658134510-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781859658134510-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 09:04:58.655508 21163 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 09:04:58.655701 21163 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 09:04:58.655723 21163 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 09:04:58.656872 21163 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 09:04:58.656916 21163 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 09:04:58.656929 21163 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 09:04:58.656940 21163 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 09:04:58.656950 21163 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 09:04:58.658368 21163 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41433
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781859658134510-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781859658134510-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40481
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781859658134510-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40481
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781859658134510-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21163
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781859658134510-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 09:04:58.658576 21163 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 09:04:58.658779 21163 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 09:04:58.663221 21168 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:04:58.663359 21163 server_base.cc:1034] running on GCE node
W20260619 09:04:58.663252 21171 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:04:58.663280 21169 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:04:58.663646 21163 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 09:04:58.663916 21163 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 09:04:58.665081 21163 hybrid_clock.cc:648] HybridClock initialized: now 1781859898665064 us; error 26 us; skew 500 ppm
Jun 19 09:04:58 dist-test-slave-mtdr krb5kdc[21150](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859898, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 09:04:58.667517 21163 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 09:04:58.667923 21163 webserver.cc:460] Webserver started at http://127.0.0.1:34225/ using document root <none> and password file <none>
I20260619 09:04:58.668109 21163 fs_manager.cc:359] Metadata directory not provided
I20260619 09:04:58.668164 21163 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 09:04:58.668280 21163 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 09:04:58.669318 21163 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781859658134510-392-0/minicluster-data/master-0/data/instance:
uuid: "d86918b6d8154b1f99e4619a85e07296"
format_stamp: "Formatted at 2026-06-19 09:04:58 on dist-test-slave-mtdr"
server_key: "5a10a6740a290f17448f64bf4e463c9a"
server_key_iv: "68f82d68b65e244918140e9d221f3dd0"
server_key_version: "encryptionkey@0"
I20260619 09:04:58.669641 21163 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781859658134510-392-0/minicluster-data/master-0/wal/instance:
uuid: "d86918b6d8154b1f99e4619a85e07296"
format_stamp: "Formatted at 2026-06-19 09:04:58 on dist-test-slave-mtdr"
server_key: "5a10a6740a290f17448f64bf4e463c9a"
server_key_iv: "68f82d68b65e244918140e9d221f3dd0"
server_key_version: "encryptionkey@0"
I20260619 09:04:58.670925 21163 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 09:04:58.671749 21178 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 09:04:58.671919 21163 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 09:04:58.672027 21163 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781859658134510-392-0/minicluster-data/master-0/data,/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781859658134510-392-0/minicluster-data/master-0/wal
uuid: "d86918b6d8154b1f99e4619a85e07296"
format_stamp: "Formatted at 2026-06-19 09:04:58 on dist-test-slave-mtdr"
server_key: "5a10a6740a290f17448f64bf4e463c9a"
server_key_iv: "68f82d68b65e244918140e9d221f3dd0"
server_key_version: "encryptionkey@0"
I20260619 09:04:58.672093 21163 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781859658134510-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781859658134510-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781859658134510-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 09:04:58.675362 21163 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 09:04:58.677213 21163 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 09:04:58.677347 21163 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 09:04:58.681226 21163 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40481
I20260619 09:04:58.681262 21230 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40481 every 8 connection(s)
I20260619 09:04:58.681545 21163 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781859658134510-392-0/minicluster-data/master-0/data/info.pb
I20260619 09:04:58.682293 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ez5NB/build/release/bin/kudu as pid 21163
I20260619 09:04:58.682410 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781859658134510-392-0/minicluster-data/master-0/wal/instance
I20260619 09:04:58.682572 21231 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 09:04:58.682590 392 external_mini_cluster.cc:1428] Setting key 703a8c5e2003253d6ea54e95646c16b0
I20260619 09:04:58.685068 21234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:04:58.683095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39213 (local address 127.0.0.1:40481)
0619 09:04:58.683269 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0619 09:04:58.683277 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 09:04:58.683303 (+ 26us) server_negotiation.cc:407] Connection header received
0619 09:04:58.683961 (+ 658us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:04:58.683970 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:04:58.684007 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:04:58.684161 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:04:58.684726 (+ 565us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:04:58.684729 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:04:58.684729 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 09:04:58.684733 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:04:58.684768 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 09:04:58.684778 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:04:58.684835 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0619 09:04:58.684908 (+ 73us) server_negotiation.cc:299] Negotiation successful
0619 09:04:58.684938 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":76,"threads_started":1}
I20260619 09:04:58.685643 21231 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d86918b6d8154b1f99e4619a85e07296: Bootstrap starting.
I20260619 09:04:58.685993 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 09:04:58.686272 21231 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d86918b6d8154b1f99e4619a85e07296: Neither blocks nor log segments found. Creating new log.
I20260619 09:04:58.686550 21231 log.cc:824] T 00000000000000000000000000000000 P d86918b6d8154b1f99e4619a85e07296: Log is configured to *not* fsync() on all Append() calls
I20260619 09:04:58.687315 21231 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d86918b6d8154b1f99e4619a85e07296: No bootstrap required, opened a new log
I20260619 09:04:58.688401 21234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:04:58.687499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33455 (local address 127.0.0.1:40481)
0619 09:04:58.687535 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0619 09:04:58.687537 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:04:58.687594 (+ 57us) server_negotiation.cc:407] Connection header received
0619 09:04:58.687650 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:04:58.687651 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:04:58.687654 (+ 3us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:04:58.687697 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:04:58.688204 (+ 507us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:04:58.688206 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:04:58.688206 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 09:04:58.688209 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:04:58.688222 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 09:04:58.688226 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:04:58.688275 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0619 09:04:58.688337 (+ 62us) server_negotiation.cc:299] Negotiation successful
0619 09:04:58.688347 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260619 09:04:58.689332 21231 raft_consensus.cc:348] T 00000000000000000000000000000000 P d86918b6d8154b1f99e4619a85e07296 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d86918b6d8154b1f99e4619a85e07296" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40481 } }
I20260619 09:04:58.689461 21231 raft_consensus.cc:374] T 00000000000000000000000000000000 P d86918b6d8154b1f99e4619a85e07296 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 09:04:58.689481 21231 raft_consensus.cc:729] T 00000000000000000000000000000000 P d86918b6d8154b1f99e4619a85e07296 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d86918b6d8154b1f99e4619a85e07296, State: Initialized, Role: FOLLOWER
I20260619 09:04:58.689579 21231 consensus_queue.cc:260] T 00000000000000000000000000000000 P d86918b6d8154b1f99e4619a85e07296 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d86918b6d8154b1f99e4619a85e07296" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40481 } }
I20260619 09:04:58.689657 21231 raft_consensus.cc:388] T 00000000000000000000000000000000 P d86918b6d8154b1f99e4619a85e07296 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 09:04:58.689680 21231 raft_consensus.cc:482] T 00000000000000000000000000000000 P d86918b6d8154b1f99e4619a85e07296 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 09:04:58.689714 21231 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d86918b6d8154b1f99e4619a85e07296 [term 0 FOLLOWER]: Advancing to term 1
I20260619 09:04:58.690191 21231 raft_consensus.cc:504] T 00000000000000000000000000000000 P d86918b6d8154b1f99e4619a85e07296 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d86918b6d8154b1f99e4619a85e07296" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40481 } }
I20260619 09:04:58.690274 21231 leader_election.cc:302] T 00000000000000000000000000000000 P d86918b6d8154b1f99e4619a85e07296 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d86918b6d8154b1f99e4619a85e07296; no voters:
I20260619 09:04:58.690423 21231 leader_election.cc:288] T 00000000000000000000000000000000 P d86918b6d8154b1f99e4619a85e07296 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 09:04:58.690455 21241 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d86918b6d8154b1f99e4619a85e07296 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 09:04:58.690594 21241 raft_consensus.cc:686] T 00000000000000000000000000000000 P d86918b6d8154b1f99e4619a85e07296 [term 1 LEADER]: Becoming Leader. State: Replica: d86918b6d8154b1f99e4619a85e07296, State: Running, Role: LEADER
I20260619 09:04:58.690625 21231 sys_catalog.cc:564] T 00000000000000000000000000000000 P d86918b6d8154b1f99e4619a85e07296 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 09:04:58.690725 21241 consensus_queue.cc:237] T 00000000000000000000000000000000 P d86918b6d8154b1f99e4619a85e07296 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d86918b6d8154b1f99e4619a85e07296" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40481 } }
I20260619 09:04:58.691059 21242 sys_catalog.cc:455] T 00000000000000000000000000000000 P d86918b6d8154b1f99e4619a85e07296 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d86918b6d8154b1f99e4619a85e07296" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d86918b6d8154b1f99e4619a85e07296" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40481 } } }
I20260619 09:04:58.691095 21243 sys_catalog.cc:455] T 00000000000000000000000000000000 P d86918b6d8154b1f99e4619a85e07296 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d86918b6d8154b1f99e4619a85e07296. Latest consensus state: current_term: 1 leader_uuid: "d86918b6d8154b1f99e4619a85e07296" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d86918b6d8154b1f99e4619a85e07296" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40481 } } }
I20260619 09:04:58.691150 21242 sys_catalog.cc:458] T 00000000000000000000000000000000 P d86918b6d8154b1f99e4619a85e07296 [sys.catalog]: This master's current role is: LEADER
I20260619 09:04:58.691170 21243 sys_catalog.cc:458] T 00000000000000000000000000000000 P d86918b6d8154b1f99e4619a85e07296 [sys.catalog]: This master's current role is: LEADER
W20260619 09:04:58.692337 21257 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d86918b6d8154b1f99e4619a85e07296: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 09:04:58.692377 21257 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 09:04:58.692435 21248 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 09:04:58.692585 21248 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 09:04:58.693787 21248 catalog_manager.cc:1347] Generated new cluster ID: d30951a79d5f4d238c389eef01985abc
I20260619 09:04:58.693828 21248 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 09:04:58.708884 21248 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 09:04:58.709046 21248 catalog_manager.cc:1504] Loading token signing keys...
I20260619 09:04:58.716748 21248 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d86918b6d8154b1f99e4619a85e07296: Generated new TSK 0
I20260619 09:04:58.716917 21248 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 09:04:58.719619 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ez5NB/build/release/bin/kudu with pid 21163
2026-06-19T09:04:58Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1203 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781859658134510-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 09:04:58 dist-test-slave-mtdr krb5kdc[21262](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 09:04:58 dist-test-slave-mtdr krb5kdc[21262](info): set up 2 sockets
Jun 19 09:04:58 dist-test-slave-mtdr krb5kdc[21262](info): commencing operation
krb5kdc: starting...
W20260619 09:04:59.658507 392 mini_kdc.cc:121] Time spent starting KDC: real 0.922s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781859658134510-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781859658134510-392-0/krb5kdc/test-user.keytab.
Jun 19 09:04:59 dist-test-slave-mtdr krb5kdc[21262](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859899, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T09:04:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T09:04:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781859658134510-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781859658134510-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 09:04:59.765777 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ez5NB/build/release/bin/kudu
/tmp/dist-test-task4ez5NB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781859658134510-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781859658134510-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781859658134510-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781859658134510-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44457
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781859658134510-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44457
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781859658134510-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781859658134510-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781859658134510-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 09:04:59.845386 21275 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 09:04:59.845592 21275 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 09:04:59.845623 21275 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 09:04:59.846757 21275 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 09:04:59.846798 21275 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 09:04:59.846844 21275 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 09:04:59.846863 21275 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 09:04:59.846885 21275 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 09:04:59.848359 21275 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781859658134510-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781859658134510-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44457
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781859658134510-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44457
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781859658134510-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21275
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781859658134510-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 09:04:59.848594 21275 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 09:04:59.848786 21275 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 09:04:59.853761 21281 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:04:59.853802 21280 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:04:59.853835 21275 server_base.cc:1034] running on GCE node
W20260619 09:04:59.853802 21283 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:04:59.854233 21275 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 09:04:59.854506 21275 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 09:04:59.855636 21275 hybrid_clock.cc:648] HybridClock initialized: now 1781859899855611 us; error 43 us; skew 500 ppm
Jun 19 09:04:59 dist-test-slave-mtdr krb5kdc[21262](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859899, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 09:04:59.858011 21275 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 09:04:59.858451 21275 webserver.cc:460] Webserver started at http://127.0.0.1:40511/ using document root <none> and password file <none>
I20260619 09:04:59.858652 21275 fs_manager.cc:359] Metadata directory not provided
I20260619 09:04:59.858700 21275 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 09:04:59.858850 21275 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 09:04:59.859874 21275 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781859658134510-392-0/minicluster-data/master-0/data/instance:
uuid: "69591000a71345358e36a6c834f1f807"
format_stamp: "Formatted at 2026-06-19 09:04:59 on dist-test-slave-mtdr"
server_key: "dbdeaef47b50412aa84095f64598f246"
server_key_iv: "d20c95b52ddb68bba91b06aaefcf55c4"
server_key_version: "encryptionkey@0"
I20260619 09:04:59.860206 21275 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781859658134510-392-0/minicluster-data/master-0/wal/instance:
uuid: "69591000a71345358e36a6c834f1f807"
format_stamp: "Formatted at 2026-06-19 09:04:59 on dist-test-slave-mtdr"
server_key: "dbdeaef47b50412aa84095f64598f246"
server_key_iv: "d20c95b52ddb68bba91b06aaefcf55c4"
server_key_version: "encryptionkey@0"
I20260619 09:04:59.861472 21275 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260619 09:04:59.862283 21290 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 09:04:59.862495 21275 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260619 09:04:59.862586 21275 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781859658134510-392-0/minicluster-data/master-0/data,/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781859658134510-392-0/minicluster-data/master-0/wal
uuid: "69591000a71345358e36a6c834f1f807"
format_stamp: "Formatted at 2026-06-19 09:04:59 on dist-test-slave-mtdr"
server_key: "dbdeaef47b50412aa84095f64598f246"
server_key_iv: "d20c95b52ddb68bba91b06aaefcf55c4"
server_key_version: "encryptionkey@0"
I20260619 09:04:59.862650 21275 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781859658134510-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781859658134510-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781859658134510-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 09:04:59.882072 21275 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 09:04:59.884256 21275 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 09:04:59.884392 21275 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 09:04:59.888305 21275 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44457
I20260619 09:04:59.888351 21342 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44457 every 8 connection(s)
I20260619 09:04:59.888676 21275 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781859658134510-392-0/minicluster-data/master-0/data/info.pb
I20260619 09:04:59.889509 21343 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 09:04:59.890100 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ez5NB/build/release/bin/kudu as pid 21275
I20260619 09:04:59.890213 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781859658134510-392-0/minicluster-data/master-0/wal/instance
I20260619 09:04:59.890412 392 external_mini_cluster.cc:1428] Setting key f1f484de517a6b00826abfdc6fb2d86c
I20260619 09:04:59.892089 21343 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 69591000a71345358e36a6c834f1f807: Bootstrap starting.
I20260619 09:04:59.892628 21343 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 69591000a71345358e36a6c834f1f807: Neither blocks nor log segments found. Creating new log.
I20260619 09:04:59.892882 21343 log.cc:824] T 00000000000000000000000000000000 P 69591000a71345358e36a6c834f1f807: Log is configured to *not* fsync() on all Append() calls
I20260619 09:04:59.893466 21343 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 69591000a71345358e36a6c834f1f807: No bootstrap required, opened a new log
I20260619 09:04:59.895210 21343 raft_consensus.cc:348] T 00000000000000000000000000000000 P 69591000a71345358e36a6c834f1f807 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69591000a71345358e36a6c834f1f807" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44457 } }
I20260619 09:04:59.895339 21343 raft_consensus.cc:374] T 00000000000000000000000000000000 P 69591000a71345358e36a6c834f1f807 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 09:04:59.895370 21343 raft_consensus.cc:729] T 00000000000000000000000000000000 P 69591000a71345358e36a6c834f1f807 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69591000a71345358e36a6c834f1f807, State: Initialized, Role: FOLLOWER
I20260619 09:04:59.895490 21343 consensus_queue.cc:260] T 00000000000000000000000000000000 P 69591000a71345358e36a6c834f1f807 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69591000a71345358e36a6c834f1f807" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44457 } }
I20260619 09:04:59.895560 21343 raft_consensus.cc:388] T 00000000000000000000000000000000 P 69591000a71345358e36a6c834f1f807 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 09:04:59.895591 21343 raft_consensus.cc:482] T 00000000000000000000000000000000 P 69591000a71345358e36a6c834f1f807 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 09:04:59.895715 21343 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 69591000a71345358e36a6c834f1f807 [term 0 FOLLOWER]: Advancing to term 1
Jun 19 09:04:59 dist-test-slave-mtdr krb5kdc[21262](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859899, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260619 09:04:59.896430 21343 raft_consensus.cc:504] T 00000000000000000000000000000000 P 69591000a71345358e36a6c834f1f807 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69591000a71345358e36a6c834f1f807" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44457 } }
I20260619 09:04:59.896526 21343 leader_election.cc:302] T 00000000000000000000000000000000 P 69591000a71345358e36a6c834f1f807 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 69591000a71345358e36a6c834f1f807; no voters:
I20260619 09:04:59.896693 21343 leader_election.cc:288] T 00000000000000000000000000000000 P 69591000a71345358e36a6c834f1f807 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 09:04:59.896740 21348 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 69591000a71345358e36a6c834f1f807 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 09:04:59.896901 21348 raft_consensus.cc:686] T 00000000000000000000000000000000 P 69591000a71345358e36a6c834f1f807 [term 1 LEADER]: Becoming Leader. State: Replica: 69591000a71345358e36a6c834f1f807, State: Running, Role: LEADER
I20260619 09:04:59.896950 21343 sys_catalog.cc:564] T 00000000000000000000000000000000 P 69591000a71345358e36a6c834f1f807 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 09:04:59.897069 21348 consensus_queue.cc:237] T 00000000000000000000000000000000 P 69591000a71345358e36a6c834f1f807 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69591000a71345358e36a6c834f1f807" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44457 } }
I20260619 09:04:59.897584 21349 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69591000a71345358e36a6c834f1f807 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "69591000a71345358e36a6c834f1f807" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69591000a71345358e36a6c834f1f807" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44457 } } }
I20260619 09:04:59.897667 21349 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69591000a71345358e36a6c834f1f807 [sys.catalog]: This master's current role is: LEADER
I20260619 09:04:59.897857 21350 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69591000a71345358e36a6c834f1f807 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 69591000a71345358e36a6c834f1f807. Latest consensus state: current_term: 1 leader_uuid: "69591000a71345358e36a6c834f1f807" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69591000a71345358e36a6c834f1f807" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44457 } } }
I20260619 09:04:59.898094 21354 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 09:04:59.898188 21350 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69591000a71345358e36a6c834f1f807 [sys.catalog]: This master's current role is: LEADER
I20260619 09:04:59.898728 21354 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 09:04:59.900631 21354 catalog_manager.cc:1347] Generated new cluster ID: ac9781bf77654a1791289633b7d3834d
I20260619 09:04:59.900684 21354 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 09:04:59.900712 21346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:04:59.890912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.131:43049 (local address 127.0.0.1:44457)
0619 09:04:59.891086 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0619 09:04:59.891094 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 09:04:59.891117 (+ 23us) server_negotiation.cc:407] Connection header received
0619 09:04:59.891502 (+ 385us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:04:59.891508 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:04:59.891554 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:04:59.891675 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:04:59.892275 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:04:59.893256 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:04:59.894017 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:04:59.894432 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:04:59.894528 (+ 96us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:04:59.897076 (+ 2548us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:04:59.897080 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:04:59.897081 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:04:59.897093 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:04:59.898992 (+ 1899us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:04:59.899456 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:04:59.899459 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:04:59.899460 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:04:59.899571 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:04:59.899827 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:04:59.899828 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:04:59.899829 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:04:59.900105 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:04:59.900254 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0619 09:04:59.900406 (+ 152us) server_negotiation.cc:299] Negotiation successful
0619 09:04:59.900513 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":78,"threads_started":1}
I20260619 09:04:59.901610 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 09:04:59.906150 21354 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 09:04:59.906834 21354 catalog_manager.cc:1504] Loading token signing keys...
I20260619 09:04:59.911154 21346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:04:59.903273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.131:36391 (local address 127.0.0.1:44457)
0619 09:04:59.903329 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0619 09:04:59.903331 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:04:59.905722 (+ 2391us) server_negotiation.cc:407] Connection header received
0619 09:04:59.905753 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:04:59.905754 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:04:59.905764 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:04:59.905835 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:04:59.906377 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:04:59.906885 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:04:59.907563 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:04:59.907770 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:04:59.907828 (+ 58us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:04:59.908887 (+ 1059us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:04:59.908888 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:04:59.908888 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:04:59.908895 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:04:59.910097 (+ 1202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:04:59.910436 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:04:59.910437 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:04:59.910438 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:04:59.910491 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:04:59.910672 (+ 181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:04:59.910673 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:04:59.910673 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:04:59.910772 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:04:59.910912 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0619 09:04:59.911017 (+ 105us) server_negotiation.cc:299] Negotiation successful
0619 09:04:59.911072 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260619 09:04:59.913604 21354 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 69591000a71345358e36a6c834f1f807: Generated new TSK 0
I20260619 09:04:59.913781 21354 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 09:04:59.915541 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ez5NB/build/release/bin/kudu with pid 21275
W20260619 09:04:59.919582 21260 connection.cc:535] client connection to 127.0.0.1:44457 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44457): Connection reset by peer (error 104)
2026-06-19T09:04:59Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1196 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781859658134510-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 09:04:59 dist-test-slave-mtdr krb5kdc[21374](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 09:04:59 dist-test-slave-mtdr krb5kdc[21374](info): set up 2 sockets
Jun 19 09:04:59 dist-test-slave-mtdr krb5kdc[21374](info): commencing operation
krb5kdc: starting...
W20260619 09:05:00.874631 392 mini_kdc.cc:121] Time spent starting KDC: real 0.942s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781859658134510-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781859658134510-392-0/krb5kdc/test-user.keytab.
Jun 19 09:05:00 dist-test-slave-mtdr krb5kdc[21374](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859900, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T09:05:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T09:05:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781859658134510-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781859658134510-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 09:05:00.985574 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ez5NB/build/release/bin/kudu
/tmp/dist-test-task4ez5NB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781859658134510-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781859658134510-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781859658134510-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781859658134510-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43389
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781859658134510-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43389
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781859658134510-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781859658134510-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781859658134510-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 09:05:01.067693 21387 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 09:05:01.067903 21387 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 09:05:01.067935 21387 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 09:05:01.069051 21387 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 09:05:01.069092 21387 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 09:05:01.069110 21387 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 09:05:01.069132 21387 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 09:05:01.069155 21387 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 09:05:01.070585 21387 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781859658134510-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781859658134510-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43389
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781859658134510-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43389
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781859658134510-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21387
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781859658134510-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 09:05:01.070806 21387 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 09:05:01.071058 21387 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 09:05:01.075562 21392 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:05:01.075558 21393 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:05:01.075721 21387 server_base.cc:1034] running on GCE node
W20260619 09:05:01.075580 21395 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:05:01.076025 21387 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 09:05:01.076293 21387 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 09:05:01.077433 21387 hybrid_clock.cc:648] HybridClock initialized: now 1781859901077421 us; error 29 us; skew 500 ppm
Jun 19 09:05:01 dist-test-slave-mtdr krb5kdc[21374](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859901, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 09:05:01.079789 21387 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 09:05:01.080221 21387 webserver.cc:460] Webserver started at http://127.0.0.1:41657/ using document root <none> and password file <none>
I20260619 09:05:01.080401 21387 fs_manager.cc:359] Metadata directory not provided
I20260619 09:05:01.080440 21387 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 09:05:01.080528 21387 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 09:05:01.081488 21387 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781859658134510-392-0/minicluster-data/master-0/data/instance:
uuid: "f8b9ee3436d6467dab07f97ce5d188ef"
format_stamp: "Formatted at 2026-06-19 09:05:01 on dist-test-slave-mtdr"
server_key: "7e66e1d5f0843bd3e3b8c30602ba54ef"
server_key_iv: "309ada1d06259641f16374097ce08807"
server_key_version: "encryptionkey@0"
I20260619 09:05:01.081759 21387 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781859658134510-392-0/minicluster-data/master-0/wal/instance:
uuid: "f8b9ee3436d6467dab07f97ce5d188ef"
format_stamp: "Formatted at 2026-06-19 09:05:01 on dist-test-slave-mtdr"
server_key: "7e66e1d5f0843bd3e3b8c30602ba54ef"
server_key_iv: "309ada1d06259641f16374097ce08807"
server_key_version: "encryptionkey@0"
I20260619 09:05:01.082955 21387 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 09:05:01.083748 21402 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 09:05:01.083940 21387 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 09:05:01.084033 21387 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781859658134510-392-0/minicluster-data/master-0/data,/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781859658134510-392-0/minicluster-data/master-0/wal
uuid: "f8b9ee3436d6467dab07f97ce5d188ef"
format_stamp: "Formatted at 2026-06-19 09:05:01 on dist-test-slave-mtdr"
server_key: "7e66e1d5f0843bd3e3b8c30602ba54ef"
server_key_iv: "309ada1d06259641f16374097ce08807"
server_key_version: "encryptionkey@0"
I20260619 09:05:01.084098 21387 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781859658134510-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781859658134510-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781859658134510-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 09:05:01.100548 21387 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 09:05:01.102619 21387 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 09:05:01.102754 21387 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 09:05:01.106415 21387 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43389
I20260619 09:05:01.106459 21454 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43389 every 8 connection(s)
I20260619 09:05:01.106770 21387 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781859658134510-392-0/minicluster-data/master-0/data/info.pb
I20260619 09:05:01.107656 21455 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 09:05:01.109853 21455 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f8b9ee3436d6467dab07f97ce5d188ef: Bootstrap starting.
I20260619 09:05:01.110289 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ez5NB/build/release/bin/kudu as pid 21387
I20260619 09:05:01.110414 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781859658134510-392-0/minicluster-data/master-0/wal/instance
I20260619 09:05:01.110533 21455 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f8b9ee3436d6467dab07f97ce5d188ef: Neither blocks nor log segments found. Creating new log.
I20260619 09:05:01.110608 392 external_mini_cluster.cc:1428] Setting key 544ccbffdaae11f9c992e92c28907ec5
I20260619 09:05:01.110872 21455 log.cc:824] T 00000000000000000000000000000000 P f8b9ee3436d6467dab07f97ce5d188ef: Log is configured to *not* fsync() on all Append() calls
I20260619 09:05:01.111586 21455 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f8b9ee3436d6467dab07f97ce5d188ef: No bootstrap required, opened a new log
I20260619 09:05:01.113489 21455 raft_consensus.cc:348] T 00000000000000000000000000000000 P f8b9ee3436d6467dab07f97ce5d188ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8b9ee3436d6467dab07f97ce5d188ef" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43389 } }
I20260619 09:05:01.113619 21455 raft_consensus.cc:374] T 00000000000000000000000000000000 P f8b9ee3436d6467dab07f97ce5d188ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 09:05:01.113651 21455 raft_consensus.cc:729] T 00000000000000000000000000000000 P f8b9ee3436d6467dab07f97ce5d188ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8b9ee3436d6467dab07f97ce5d188ef, State: Initialized, Role: FOLLOWER
I20260619 09:05:01.113770 21455 consensus_queue.cc:260] T 00000000000000000000000000000000 P f8b9ee3436d6467dab07f97ce5d188ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8b9ee3436d6467dab07f97ce5d188ef" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43389 } }
I20260619 09:05:01.113843 21455 raft_consensus.cc:388] T 00000000000000000000000000000000 P f8b9ee3436d6467dab07f97ce5d188ef [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 09:05:01.113888 21455 raft_consensus.cc:482] T 00000000000000000000000000000000 P f8b9ee3436d6467dab07f97ce5d188ef [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 09:05:01.113934 21455 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f8b9ee3436d6467dab07f97ce5d188ef [term 0 FOLLOWER]: Advancing to term 1
I20260619 09:05:01.114434 21455 raft_consensus.cc:504] T 00000000000000000000000000000000 P f8b9ee3436d6467dab07f97ce5d188ef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8b9ee3436d6467dab07f97ce5d188ef" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43389 } }
I20260619 09:05:01.114526 21455 leader_election.cc:302] T 00000000000000000000000000000000 P f8b9ee3436d6467dab07f97ce5d188ef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f8b9ee3436d6467dab07f97ce5d188ef; no voters:
I20260619 09:05:01.114688 21455 leader_election.cc:288] T 00000000000000000000000000000000 P f8b9ee3436d6467dab07f97ce5d188ef [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 09:05:01.114723 21460 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f8b9ee3436d6467dab07f97ce5d188ef [term 1 FOLLOWER]: Leader election won for term 1
I20260619 09:05:01.114842 21460 raft_consensus.cc:686] T 00000000000000000000000000000000 P f8b9ee3436d6467dab07f97ce5d188ef [term 1 LEADER]: Becoming Leader. State: Replica: f8b9ee3436d6467dab07f97ce5d188ef, State: Running, Role: LEADER
I20260619 09:05:01.114917 21460 consensus_queue.cc:237] T 00000000000000000000000000000000 P f8b9ee3436d6467dab07f97ce5d188ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8b9ee3436d6467dab07f97ce5d188ef" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43389 } }
I20260619 09:05:01.115000 21455 sys_catalog.cc:564] T 00000000000000000000000000000000 P f8b9ee3436d6467dab07f97ce5d188ef [sys.catalog]: configured and running, proceeding with master startup.
I20260619 09:05:01.115192 21460 sys_catalog.cc:455] T 00000000000000000000000000000000 P f8b9ee3436d6467dab07f97ce5d188ef [sys.catalog]: SysCatalogTable state changed. Reason: New leader f8b9ee3436d6467dab07f97ce5d188ef. Latest consensus state: current_term: 1 leader_uuid: "f8b9ee3436d6467dab07f97ce5d188ef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8b9ee3436d6467dab07f97ce5d188ef" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43389 } } }
I20260619 09:05:01.115257 21460 sys_catalog.cc:458] T 00000000000000000000000000000000 P f8b9ee3436d6467dab07f97ce5d188ef [sys.catalog]: This master's current role is: LEADER
I20260619 09:05:01.115223 21461 sys_catalog.cc:455] T 00000000000000000000000000000000 P f8b9ee3436d6467dab07f97ce5d188ef [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f8b9ee3436d6467dab07f97ce5d188ef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8b9ee3436d6467dab07f97ce5d188ef" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43389 } } }
I20260619 09:05:01.115298 21461 sys_catalog.cc:458] T 00000000000000000000000000000000 P f8b9ee3436d6467dab07f97ce5d188ef [sys.catalog]: This master's current role is: LEADER
I20260619 09:05:01.115944 21459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:05:01.111335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.131:33957 (local address 127.0.0.1:43389)
0619 09:05:01.111487 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0619 09:05:01.111496 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 09:05:01.111520 (+ 24us) server_negotiation.cc:407] Connection header received
0619 09:05:01.111945 (+ 425us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:05:01.111952 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:05:01.112001 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:05:01.112126 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:05:01.112349 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:05:01.113204 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:05:01.114011 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:05:01.114407 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:05:01.114525 (+ 118us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:05:01.115200 (+ 675us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:05:01.115204 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:05:01.115205 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 09:05:01.115491 (+ 286us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:05:01.115522 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 09:05:01.115533 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:05:01.115604 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0619 09:05:01.115723 (+ 119us) server_negotiation.cc:299] Negotiation successful
0619 09:05:01.115751 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101,"thread_start_us":65,"threads_started":1}
I20260619 09:05:01.116096 21464 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 09:05:01.116534 21464 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 09:05:01.117017 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 09:05:01.118211 21464 catalog_manager.cc:1347] Generated new cluster ID: 234842ba54d1441686d6573a5cc44758
I20260619 09:05:01.118260 21464 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 09:05:01.121842 21459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:05:01.118474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.131:57771 (local address 127.0.0.1:43389)
0619 09:05:01.118508 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0619 09:05:01.118509 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 09:05:01.118650 (+ 141us) server_negotiation.cc:407] Connection header received
0619 09:05:01.118710 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:05:01.118711 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:05:01.118722 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:05:01.118777 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:05:01.118922 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:05:01.119648 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:05:01.120590 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:05:01.120922 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:05:01.120990 (+ 68us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:05:01.121609 (+ 619us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:05:01.121611 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:05:01.121611 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 09:05:01.121615 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:05:01.121631 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 09:05:01.121635 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:05:01.121692 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0619 09:05:01.121758 (+ 66us) server_negotiation.cc:299] Negotiation successful
0619 09:05:01.121768 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260619 09:05:01.125468 21464 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 09:05:01.125949 21464 catalog_manager.cc:1504] Loading token signing keys...
I20260619 09:05:01.137904 21464 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f8b9ee3436d6467dab07f97ce5d188ef: Generated new TSK 0
I20260619 09:05:01.138087 21464 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 09:05:01.141853 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ez5NB/build/release/bin/kudu with pid 21387
W20260619 09:05:01.145680 21372 connection.cc:535] client connection to 127.0.0.1:43389 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43389): Connection reset by peer (error 104)
2026-06-19T09:05:01Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1226 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781859658134510-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 09:05:01 dist-test-slave-mtdr krb5kdc[21486](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 09:05:01 dist-test-slave-mtdr krb5kdc[21486](info): set up 2 sockets
Jun 19 09:05:01 dist-test-slave-mtdr krb5kdc[21486](info): commencing operation
krb5kdc: starting...
W20260619 09:05:02.090890 392 mini_kdc.cc:121] Time spent starting KDC: real 0.932s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781859658134510-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781859658134510-392-0/krb5kdc/test-user.keytab.
Jun 19 09:05:02 dist-test-slave-mtdr krb5kdc[21486](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859902, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T09:05:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T09:05:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781859658134510-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781859658134510-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 09:05:02.205542 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ez5NB/build/release/bin/kudu
/tmp/dist-test-task4ez5NB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781859658134510-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781859658134510-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781859658134510-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781859658134510-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33875
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781859658134510-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33875
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781859658134510-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781859658134510-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781859658134510-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 09:05:02.285413 21499 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 09:05:02.285633 21499 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 09:05:02.285667 21499 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 09:05:02.287101 21499 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 09:05:02.287153 21499 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 09:05:02.287174 21499 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 09:05:02.287191 21499 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 09:05:02.287205 21499 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 09:05:02.288862 21499 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781859658134510-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781859658134510-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33875
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781859658134510-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33875
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781859658134510-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21499
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781859658134510-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 09:05:02.289108 21499 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 09:05:02.289319 21499 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 09:05:02.293949 21507 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:05:02.294003 21505 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:05:02.294021 21499 server_base.cc:1034] running on GCE node
W20260619 09:05:02.293951 21504 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:05:02.294412 21499 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 09:05:02.294672 21499 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 09:05:02.295835 21499 hybrid_clock.cc:648] HybridClock initialized: now 1781859902295808 us; error 40 us; skew 500 ppm
Jun 19 09:05:02 dist-test-slave-mtdr krb5kdc[21486](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859902, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 09:05:02.298380 21499 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 09:05:02.298977 21499 webserver.cc:460] Webserver started at http://127.0.0.1:34525/ using document root <none> and password file <none>
I20260619 09:05:02.299211 21499 fs_manager.cc:359] Metadata directory not provided
I20260619 09:05:02.299276 21499 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 09:05:02.299405 21499 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 09:05:02.300427 21499 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781859658134510-392-0/minicluster-data/master-0/data/instance:
uuid: "cac00829c71041f78456f7db41659b65"
format_stamp: "Formatted at 2026-06-19 09:05:02 on dist-test-slave-mtdr"
server_key: "086ba234a7f922db726b7f7f63b904d1"
server_key_iv: "1cbf2bbcd89521b2d939aa6cce074c33"
server_key_version: "encryptionkey@0"
I20260619 09:05:02.300758 21499 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781859658134510-392-0/minicluster-data/master-0/wal/instance:
uuid: "cac00829c71041f78456f7db41659b65"
format_stamp: "Formatted at 2026-06-19 09:05:02 on dist-test-slave-mtdr"
server_key: "086ba234a7f922db726b7f7f63b904d1"
server_key_iv: "1cbf2bbcd89521b2d939aa6cce074c33"
server_key_version: "encryptionkey@0"
I20260619 09:05:02.302032 21499 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 09:05:02.302867 21514 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 09:05:02.303105 21499 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260619 09:05:02.303198 21499 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781859658134510-392-0/minicluster-data/master-0/data,/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781859658134510-392-0/minicluster-data/master-0/wal
uuid: "cac00829c71041f78456f7db41659b65"
format_stamp: "Formatted at 2026-06-19 09:05:02 on dist-test-slave-mtdr"
server_key: "086ba234a7f922db726b7f7f63b904d1"
server_key_iv: "1cbf2bbcd89521b2d939aa6cce074c33"
server_key_version: "encryptionkey@0"
I20260619 09:05:02.303269 21499 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781859658134510-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781859658134510-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781859658134510-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 09:05:02.306458 21499 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 09:05:02.308367 21499 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 09:05:02.308498 21499 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 09:05:02.312364 21499 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33875
I20260619 09:05:02.312397 21566 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33875 every 8 connection(s)
I20260619 09:05:02.312780 21499 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781859658134510-392-0/minicluster-data/master-0/data/info.pb
I20260619 09:05:02.313733 21567 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 09:05:02.316226 21567 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cac00829c71041f78456f7db41659b65: Bootstrap starting.
I20260619 09:05:02.316802 21567 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cac00829c71041f78456f7db41659b65: Neither blocks nor log segments found. Creating new log.
I20260619 09:05:02.317061 21567 log.cc:824] T 00000000000000000000000000000000 P cac00829c71041f78456f7db41659b65: Log is configured to *not* fsync() on all Append() calls
I20260619 09:05:02.317700 21567 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cac00829c71041f78456f7db41659b65: No bootstrap required, opened a new log
I20260619 09:05:02.319264 21567 raft_consensus.cc:348] T 00000000000000000000000000000000 P cac00829c71041f78456f7db41659b65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cac00829c71041f78456f7db41659b65" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33875 } }
I20260619 09:05:02.319411 21567 raft_consensus.cc:374] T 00000000000000000000000000000000 P cac00829c71041f78456f7db41659b65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 09:05:02.319433 21567 raft_consensus.cc:729] T 00000000000000000000000000000000 P cac00829c71041f78456f7db41659b65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cac00829c71041f78456f7db41659b65, State: Initialized, Role: FOLLOWER
I20260619 09:05:02.319555 21567 consensus_queue.cc:260] T 00000000000000000000000000000000 P cac00829c71041f78456f7db41659b65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cac00829c71041f78456f7db41659b65" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33875 } }
I20260619 09:05:02.319612 21567 raft_consensus.cc:388] T 00000000000000000000000000000000 P cac00829c71041f78456f7db41659b65 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 09:05:02.319633 21567 raft_consensus.cc:482] T 00000000000000000000000000000000 P cac00829c71041f78456f7db41659b65 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 09:05:02.319659 21567 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cac00829c71041f78456f7db41659b65 [term 0 FOLLOWER]: Advancing to term 1
I20260619 09:05:02.320145 21567 raft_consensus.cc:504] T 00000000000000000000000000000000 P cac00829c71041f78456f7db41659b65 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cac00829c71041f78456f7db41659b65" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33875 } }
I20260619 09:05:02.320188 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ez5NB/build/release/bin/kudu as pid 21499
I20260619 09:05:02.320240 21567 leader_election.cc:302] T 00000000000000000000000000000000 P cac00829c71041f78456f7db41659b65 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cac00829c71041f78456f7db41659b65; no voters:
I20260619 09:05:02.320283 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781859658134510-392-0/minicluster-data/master-0/wal/instance
I20260619 09:05:02.320399 21567 leader_election.cc:288] T 00000000000000000000000000000000 P cac00829c71041f78456f7db41659b65 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 09:05:02.320436 21570 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cac00829c71041f78456f7db41659b65 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 09:05:02.320482 392 external_mini_cluster.cc:1428] Setting key 2241881e8dd308f15841555549932efb
I20260619 09:05:02.320544 21570 raft_consensus.cc:686] T 00000000000000000000000000000000 P cac00829c71041f78456f7db41659b65 [term 1 LEADER]: Becoming Leader. State: Replica: cac00829c71041f78456f7db41659b65, State: Running, Role: LEADER
I20260619 09:05:02.320611 21570 consensus_queue.cc:237] T 00000000000000000000000000000000 P cac00829c71041f78456f7db41659b65 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cac00829c71041f78456f7db41659b65" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33875 } }
I20260619 09:05:02.320644 21567 sys_catalog.cc:564] T 00000000000000000000000000000000 P cac00829c71041f78456f7db41659b65 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 09:05:02.320950 21570 sys_catalog.cc:455] T 00000000000000000000000000000000 P cac00829c71041f78456f7db41659b65 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cac00829c71041f78456f7db41659b65. Latest consensus state: current_term: 1 leader_uuid: "cac00829c71041f78456f7db41659b65" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cac00829c71041f78456f7db41659b65" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33875 } } }
I20260619 09:05:02.321012 21570 sys_catalog.cc:458] T 00000000000000000000000000000000 P cac00829c71041f78456f7db41659b65 [sys.catalog]: This master's current role is: LEADER
I20260619 09:05:02.321187 21571 sys_catalog.cc:455] T 00000000000000000000000000000000 P cac00829c71041f78456f7db41659b65 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cac00829c71041f78456f7db41659b65" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cac00829c71041f78456f7db41659b65" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33875 } } }
I20260619 09:05:02.321264 21571 sys_catalog.cc:458] T 00000000000000000000000000000000 P cac00829c71041f78456f7db41659b65 [sys.catalog]: This master's current role is: LEADER
I20260619 09:05:02.321452 21575 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 09:05:02.322369 21575 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 09:05:02.322875 21576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:05:02.321023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.131:49585 (local address 127.0.0.1:33875)
0619 09:05:02.321217 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0619 09:05:02.321226 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 09:05:02.321257 (+ 31us) server_negotiation.cc:407] Connection header received
0619 09:05:02.321389 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:05:02.321396 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:05:02.321429 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:05:02.321609 (+ 180us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:05:02.322283 (+ 674us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:05:02.322287 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:05:02.322287 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 09:05:02.322292 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:05:02.322328 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 09:05:02.322342 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:05:02.322407 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0619 09:05:02.322640 (+ 233us) server_negotiation.cc:299] Negotiation successful
0619 09:05:02.322683 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":87,"threads_started":1}
I20260619 09:05:02.323770 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 09:05:02.324031 21575 catalog_manager.cc:1347] Generated new cluster ID: 4ee2d7def2ca4b92810ea38b36f243a2
I20260619 09:05:02.324079 21575 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 09:05:02.326155 21576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:05:02.325172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.131:48223 (local address 127.0.0.1:33875)
0619 09:05:02.325205 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0619 09:05:02.325206 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 09:05:02.325348 (+ 142us) server_negotiation.cc:407] Connection header received
0619 09:05:02.325401 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:05:02.325401 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:05:02.325419 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:05:02.325462 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:05:02.325961 (+ 499us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:05:02.325962 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:05:02.325963 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 09:05:02.325966 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:05:02.325980 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 09:05:02.325983 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:05:02.326028 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0619 09:05:02.326076 (+ 48us) server_negotiation.cc:299] Negotiation successful
0619 09:05:02.326089 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260619 09:05:02.339246 21575 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 09:05:02.339426 21575 catalog_manager.cc:1504] Loading token signing keys...
I20260619 09:05:02.346980 21575 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cac00829c71041f78456f7db41659b65: Generated new TSK 0
I20260619 09:05:02.347160 21575 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 09:05:02.350312 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ez5NB/build/release/bin/kudu with pid 21499
2026-06-19T09:05:02Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1208 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781859658134510-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 09:05:02 dist-test-slave-mtdr krb5kdc[21598](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 09:05:02 dist-test-slave-mtdr krb5kdc[21598](info): set up 2 sockets
Jun 19 09:05:02 dist-test-slave-mtdr krb5kdc[21598](info): commencing operation
krb5kdc: starting...
W20260619 09:05:03.310741 392 mini_kdc.cc:121] Time spent starting KDC: real 0.943s user 0.001s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781859658134510-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781859658134510-392-0/krb5kdc/test-user.keytab.
Jun 19 09:05:03 dist-test-slave-mtdr krb5kdc[21598](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859903, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T09:05:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T09:05:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781859658134510-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781859658134510-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 09:05:03.421685 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ez5NB/build/release/bin/kudu
/tmp/dist-test-task4ez5NB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781859658134510-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781859658134510-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781859658134510-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781859658134510-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42229
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781859658134510-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42229
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781859658134510-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781859658134510-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781859658134510-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 09:05:03.499241 21611 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 09:05:03.499436 21611 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 09:05:03.499473 21611 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 09:05:03.500598 21611 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 09:05:03.500641 21611 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 09:05:03.500659 21611 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 09:05:03.500675 21611 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 09:05:03.500691 21611 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 09:05:03.502156 21611 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781859658134510-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781859658134510-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42229
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781859658134510-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42229
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781859658134510-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21611
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781859658134510-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 09:05:03.502384 21611 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 09:05:03.502585 21611 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 09:05:03.507072 21617 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:05:03.507090 21619 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:05:03.507079 21616 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:05:03.507079 21611 server_base.cc:1034] running on GCE node
I20260619 09:05:03.507495 21611 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 09:05:03.507750 21611 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 09:05:03.508910 21611 hybrid_clock.cc:648] HybridClock initialized: now 1781859903508897 us; error 27 us; skew 500 ppm
Jun 19 09:05:03 dist-test-slave-mtdr krb5kdc[21598](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859903, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 09:05:03.511266 21611 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 09:05:03.511678 21611 webserver.cc:460] Webserver started at http://127.0.0.1:35911/ using document root <none> and password file <none>
I20260619 09:05:03.511895 21611 fs_manager.cc:359] Metadata directory not provided
I20260619 09:05:03.511945 21611 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 09:05:03.512063 21611 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 09:05:03.513082 21611 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781859658134510-392-0/minicluster-data/master-0/data/instance:
uuid: "8b933293c9904f4eb267e87073e84eac"
format_stamp: "Formatted at 2026-06-19 09:05:03 on dist-test-slave-mtdr"
server_key: "d2f90efa651e3008a35680c1e6b0c4f9"
server_key_iv: "722cf1bd1b1de06b659fcbe401c77953"
server_key_version: "encryptionkey@0"
I20260619 09:05:03.513397 21611 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781859658134510-392-0/minicluster-data/master-0/wal/instance:
uuid: "8b933293c9904f4eb267e87073e84eac"
format_stamp: "Formatted at 2026-06-19 09:05:03 on dist-test-slave-mtdr"
server_key: "d2f90efa651e3008a35680c1e6b0c4f9"
server_key_iv: "722cf1bd1b1de06b659fcbe401c77953"
server_key_version: "encryptionkey@0"
I20260619 09:05:03.514631 21611 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260619 09:05:03.515341 21626 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 09:05:03.515486 21611 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260619 09:05:03.515565 21611 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781859658134510-392-0/minicluster-data/master-0/data,/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781859658134510-392-0/minicluster-data/master-0/wal
uuid: "8b933293c9904f4eb267e87073e84eac"
format_stamp: "Formatted at 2026-06-19 09:05:03 on dist-test-slave-mtdr"
server_key: "d2f90efa651e3008a35680c1e6b0c4f9"
server_key_iv: "722cf1bd1b1de06b659fcbe401c77953"
server_key_version: "encryptionkey@0"
I20260619 09:05:03.515631 21611 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781859658134510-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781859658134510-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781859658134510-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 09:05:03.526535 21611 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 09:05:03.528486 21611 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 09:05:03.528625 21611 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 09:05:03.532193 21611 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42229
I20260619 09:05:03.532224 21678 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42229 every 8 connection(s)
I20260619 09:05:03.532531 21611 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781859658134510-392-0/minicluster-data/master-0/data/info.pb
I20260619 09:05:03.533360 21679 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 09:05:03.535530 21679 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b933293c9904f4eb267e87073e84eac: Bootstrap starting.
I20260619 09:05:03.536113 21679 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8b933293c9904f4eb267e87073e84eac: Neither blocks nor log segments found. Creating new log.
I20260619 09:05:03.536109 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ez5NB/build/release/bin/kudu as pid 21611
I20260619 09:05:03.536211 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781859658134510-392-0/minicluster-data/master-0/wal/instance
I20260619 09:05:03.536412 392 external_mini_cluster.cc:1428] Setting key f8d324d04f341a22897caaebcc9aeed3
I20260619 09:05:03.536414 21679 log.cc:824] T 00000000000000000000000000000000 P 8b933293c9904f4eb267e87073e84eac: Log is configured to *not* fsync() on all Append() calls
I20260619 09:05:03.537134 21679 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b933293c9904f4eb267e87073e84eac: No bootstrap required, opened a new log
I20260619 09:05:03.539247 21679 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8b933293c9904f4eb267e87073e84eac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b933293c9904f4eb267e87073e84eac" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42229 } }
I20260619 09:05:03.539398 21679 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8b933293c9904f4eb267e87073e84eac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 09:05:03.539433 21679 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8b933293c9904f4eb267e87073e84eac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b933293c9904f4eb267e87073e84eac, State: Initialized, Role: FOLLOWER
I20260619 09:05:03.539561 21679 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8b933293c9904f4eb267e87073e84eac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b933293c9904f4eb267e87073e84eac" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42229 } }
I20260619 09:05:03.539625 21679 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8b933293c9904f4eb267e87073e84eac [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 09:05:03.539655 21679 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8b933293c9904f4eb267e87073e84eac [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 09:05:03.539700 21679 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8b933293c9904f4eb267e87073e84eac [term 0 FOLLOWER]: Advancing to term 1
I20260619 09:05:03.540211 21679 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8b933293c9904f4eb267e87073e84eac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b933293c9904f4eb267e87073e84eac" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42229 } }
I20260619 09:05:03.540300 21679 leader_election.cc:302] T 00000000000000000000000000000000 P 8b933293c9904f4eb267e87073e84eac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8b933293c9904f4eb267e87073e84eac; no voters:
I20260619 09:05:03.540448 21679 leader_election.cc:288] T 00000000000000000000000000000000 P 8b933293c9904f4eb267e87073e84eac [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 09:05:03.540485 21684 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8b933293c9904f4eb267e87073e84eac [term 1 FOLLOWER]: Leader election won for term 1
I20260619 09:05:03.540588 21684 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8b933293c9904f4eb267e87073e84eac [term 1 LEADER]: Becoming Leader. State: Replica: 8b933293c9904f4eb267e87073e84eac, State: Running, Role: LEADER
I20260619 09:05:03.540699 21679 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8b933293c9904f4eb267e87073e84eac [sys.catalog]: configured and running, proceeding with master startup.
I20260619 09:05:03.540716 21684 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8b933293c9904f4eb267e87073e84eac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b933293c9904f4eb267e87073e84eac" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42229 } }
I20260619 09:05:03.541071 21686 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b933293c9904f4eb267e87073e84eac [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8b933293c9904f4eb267e87073e84eac. Latest consensus state: current_term: 1 leader_uuid: "8b933293c9904f4eb267e87073e84eac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b933293c9904f4eb267e87073e84eac" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42229 } } }
I20260619 09:05:03.541188 21686 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b933293c9904f4eb267e87073e84eac [sys.catalog]: This master's current role is: LEADER
Jun 19 09:05:03 dist-test-slave-mtdr krb5kdc[21598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859903, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260619 09:05:03.542500 21685 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b933293c9904f4eb267e87073e84eac [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8b933293c9904f4eb267e87073e84eac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b933293c9904f4eb267e87073e84eac" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42229 } } }
I20260619 09:05:03.543011 21685 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b933293c9904f4eb267e87073e84eac [sys.catalog]: This master's current role is: LEADER
W20260619 09:05:03.542676 21699 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8b933293c9904f4eb267e87073e84eac: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 09:05:03.543095 21699 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 09:05:03.543159 21700 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 09:05:03.543318 21700 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 09:05:03.545197 21700 catalog_manager.cc:1347] Generated new cluster ID: 0f8e16b0b5aa4c61bcc61fb9d2df0eaa
I20260619 09:05:03.545234 21700 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 09:05:03.546630 21683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:05:03.536898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.131:43011 (local address 127.0.0.1:42229)
0619 09:05:03.537049 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0619 09:05:03.537058 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 09:05:03.537093 (+ 35us) server_negotiation.cc:407] Connection header received
0619 09:05:03.537550 (+ 457us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:05:03.537557 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:05:03.537607 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:05:03.537726 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:05:03.538349 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:05:03.539336 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:05:03.540139 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:05:03.540638 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:05:03.540739 (+ 101us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:05:03.543306 (+ 2567us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:05:03.543310 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:05:03.543310 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:05:03.543322 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:05:03.545084 (+ 1762us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:05:03.545505 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:05:03.545508 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:05:03.545508 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:05:03.545601 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:05:03.545813 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:05:03.545813 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:05:03.545814 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:05:03.546075 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:05:03.546226 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0619 09:05:03.546342 (+ 116us) server_negotiation.cc:299] Negotiation successful
0619 09:05:03.546438 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100,"thread_start_us":68,"threads_started":1}
I20260619 09:05:03.547789 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 09:05:03.557521 21683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:05:03.549289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.131:60275 (local address 127.0.0.1:42229)
0619 09:05:03.549325 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0619 09:05:03.549327 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:05:03.550987 (+ 1660us) server_negotiation.cc:407] Connection header received
0619 09:05:03.551067 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:05:03.551068 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:05:03.551078 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:05:03.551144 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:05:03.551686 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:05:03.552458 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:05:03.553396 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:05:03.553725 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:05:03.553781 (+ 56us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:05:03.555028 (+ 1247us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:05:03.555029 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:05:03.555030 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:05:03.555036 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:05:03.556452 (+ 1416us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:05:03.556820 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:05:03.556820 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:05:03.556821 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:05:03.556875 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:05:03.557060 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:05:03.557060 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:05:03.557060 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:05:03.557178 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:05:03.557275 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0619 09:05:03.557372 (+ 97us) server_negotiation.cc:299] Negotiation successful
0619 09:05:03.557432 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260619 09:05:03.565673 21700 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 09:05:03.566234 21700 catalog_manager.cc:1504] Loading token signing keys...
I20260619 09:05:03.569068 21700 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8b933293c9904f4eb267e87073e84eac: Generated new TSK 0
I20260619 09:05:03.569242 21700 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 09:05:03.579994 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ez5NB/build/release/bin/kudu with pid 21611
W20260619 09:05:03.583750 21596 connection.cc:535] client connection to 127.0.0.1:42229 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42229): Connection reset by peer (error 104)
2026-06-19T09:05:03Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1229 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781859658134510-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 09:05:03 dist-test-slave-mtdr krb5kdc[21710](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 09:05:03 dist-test-slave-mtdr krb5kdc[21710](info): set up 2 sockets
Jun 19 09:05:03 dist-test-slave-mtdr krb5kdc[21710](info): commencing operation
krb5kdc: starting...
W20260619 09:05:04.522581 392 mini_kdc.cc:121] Time spent starting KDC: real 0.925s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781859658134510-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781859658134510-392-0/krb5kdc/test-user.keytab.
Jun 19 09:05:04 dist-test-slave-mtdr krb5kdc[21710](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859904, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T09:05:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T09:05:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781859658134510-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781859658134510-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 09:05:04.633562 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ez5NB/build/release/bin/kudu
/tmp/dist-test-task4ez5NB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781859658134510-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781859658134510-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781859658134510-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781859658134510-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44277
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781859658134510-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33371
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44277
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781859658134510-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781859658134510-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781859658134510-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 09:05:04.710889 21723 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 09:05:04.711100 21723 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 09:05:04.711129 21723 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 09:05:04.712244 21723 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 09:05:04.712285 21723 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 09:05:04.712302 21723 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 09:05:04.712318 21723 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 09:05:04.712334 21723 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 09:05:04.713761 21723 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33371
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781859658134510-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781859658134510-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44277
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781859658134510-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44277
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781859658134510-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21723
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781859658134510-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 09:05:04.713997 21723 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 09:05:04.714195 21723 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 09:05:04.719103 21731 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:05:04.719115 21728 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:05:04.719172 21729 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:05:04.719363 21723 server_base.cc:1034] running on GCE node
I20260619 09:05:04.719614 21723 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 09:05:04.719879 21723 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 09:05:04.721026 21723 hybrid_clock.cc:648] HybridClock initialized: now 1781859904721003 us; error 32 us; skew 500 ppm
Jun 19 09:05:04 dist-test-slave-mtdr krb5kdc[21710](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859904, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 09:05:04.723592 21723 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 09:05:04.724030 21723 webserver.cc:460] Webserver started at http://127.0.0.1:43187/ using document root <none> and password file <none>
I20260619 09:05:04.724243 21723 fs_manager.cc:359] Metadata directory not provided
I20260619 09:05:04.724298 21723 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 09:05:04.724424 21723 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 09:05:04.725455 21723 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781859658134510-392-0/minicluster-data/master-0/data/instance:
uuid: "291000932a274845a17b7c1e5504ae81"
format_stamp: "Formatted at 2026-06-19 09:05:04 on dist-test-slave-mtdr"
server_key: "b1b460ecf85ea3e40c708bd64981ba79"
server_key_iv: "2789ab9f9c6d240cebe3cf4da0b0e554"
server_key_version: "encryptionkey@0"
I20260619 09:05:04.725805 21723 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781859658134510-392-0/minicluster-data/master-0/wal/instance:
uuid: "291000932a274845a17b7c1e5504ae81"
format_stamp: "Formatted at 2026-06-19 09:05:04 on dist-test-slave-mtdr"
server_key: "b1b460ecf85ea3e40c708bd64981ba79"
server_key_iv: "2789ab9f9c6d240cebe3cf4da0b0e554"
server_key_version: "encryptionkey@0"
I20260619 09:05:04.727111 21723 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260619 09:05:04.727912 21738 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 09:05:04.728062 21723 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 09:05:04.728159 21723 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781859658134510-392-0/minicluster-data/master-0/data,/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781859658134510-392-0/minicluster-data/master-0/wal
uuid: "291000932a274845a17b7c1e5504ae81"
format_stamp: "Formatted at 2026-06-19 09:05:04 on dist-test-slave-mtdr"
server_key: "b1b460ecf85ea3e40c708bd64981ba79"
server_key_iv: "2789ab9f9c6d240cebe3cf4da0b0e554"
server_key_version: "encryptionkey@0"
I20260619 09:05:04.728222 21723 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781859658134510-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781859658134510-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781859658134510-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 09:05:04.744948 21723 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 09:05:04.746836 21723 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 09:05:04.746981 21723 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 09:05:04.750726 21723 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44277
I20260619 09:05:04.750771 21790 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44277 every 8 connection(s)
I20260619 09:05:04.751150 21723 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781859658134510-392-0/minicluster-data/master-0/data/info.pb
I20260619 09:05:04.752132 21791 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 09:05:04.754498 21791 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 291000932a274845a17b7c1e5504ae81: Bootstrap starting.
I20260619 09:05:04.755123 21791 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 291000932a274845a17b7c1e5504ae81: Neither blocks nor log segments found. Creating new log.
I20260619 09:05:04.755388 21791 log.cc:824] T 00000000000000000000000000000000 P 291000932a274845a17b7c1e5504ae81: Log is configured to *not* fsync() on all Append() calls
I20260619 09:05:04.756109 21791 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 291000932a274845a17b7c1e5504ae81: No bootstrap required, opened a new log
I20260619 09:05:04.757656 21791 raft_consensus.cc:348] T 00000000000000000000000000000000 P 291000932a274845a17b7c1e5504ae81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "291000932a274845a17b7c1e5504ae81" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44277 } }
I20260619 09:05:04.757793 21791 raft_consensus.cc:374] T 00000000000000000000000000000000 P 291000932a274845a17b7c1e5504ae81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 09:05:04.757817 21791 raft_consensus.cc:729] T 00000000000000000000000000000000 P 291000932a274845a17b7c1e5504ae81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 291000932a274845a17b7c1e5504ae81, State: Initialized, Role: FOLLOWER
I20260619 09:05:04.757804 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ez5NB/build/release/bin/kudu as pid 21723
I20260619 09:05:04.757929 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781859658134510-392-0/minicluster-data/master-0/wal/instance
I20260619 09:05:04.757939 21791 consensus_queue.cc:260] T 00000000000000000000000000000000 P 291000932a274845a17b7c1e5504ae81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "291000932a274845a17b7c1e5504ae81" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44277 } }
I20260619 09:05:04.758004 21791 raft_consensus.cc:388] T 00000000000000000000000000000000 P 291000932a274845a17b7c1e5504ae81 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 09:05:04.758051 21791 raft_consensus.cc:482] T 00000000000000000000000000000000 P 291000932a274845a17b7c1e5504ae81 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 09:05:04.758101 21791 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 291000932a274845a17b7c1e5504ae81 [term 0 FOLLOWER]: Advancing to term 1
I20260619 09:05:04.758142 392 external_mini_cluster.cc:1428] Setting key 9b9e4ac6d27489ce265aa1fc63ab9053
I20260619 09:05:04.758694 21791 raft_consensus.cc:504] T 00000000000000000000000000000000 P 291000932a274845a17b7c1e5504ae81 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "291000932a274845a17b7c1e5504ae81" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44277 } }
I20260619 09:05:04.758834 21791 leader_election.cc:302] T 00000000000000000000000000000000 P 291000932a274845a17b7c1e5504ae81 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 291000932a274845a17b7c1e5504ae81; no voters:
I20260619 09:05:04.759024 21791 leader_election.cc:288] T 00000000000000000000000000000000 P 291000932a274845a17b7c1e5504ae81 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 09:05:04.759043 21796 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 291000932a274845a17b7c1e5504ae81 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 09:05:04.759184 21796 raft_consensus.cc:686] T 00000000000000000000000000000000 P 291000932a274845a17b7c1e5504ae81 [term 1 LEADER]: Becoming Leader. State: Replica: 291000932a274845a17b7c1e5504ae81, State: Running, Role: LEADER
I20260619 09:05:04.759346 21791 sys_catalog.cc:564] T 00000000000000000000000000000000 P 291000932a274845a17b7c1e5504ae81 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 09:05:04.759331 21796 consensus_queue.cc:237] T 00000000000000000000000000000000 P 291000932a274845a17b7c1e5504ae81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "291000932a274845a17b7c1e5504ae81" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44277 } }
I20260619 09:05:04.759753 21798 sys_catalog.cc:455] T 00000000000000000000000000000000 P 291000932a274845a17b7c1e5504ae81 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "291000932a274845a17b7c1e5504ae81" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "291000932a274845a17b7c1e5504ae81" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44277 } } }
I20260619 09:05:04.759874 21798 sys_catalog.cc:458] T 00000000000000000000000000000000 P 291000932a274845a17b7c1e5504ae81 [sys.catalog]: This master's current role is: LEADER
I20260619 09:05:04.760057 21797 sys_catalog.cc:455] T 00000000000000000000000000000000 P 291000932a274845a17b7c1e5504ae81 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 291000932a274845a17b7c1e5504ae81. Latest consensus state: current_term: 1 leader_uuid: "291000932a274845a17b7c1e5504ae81" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "291000932a274845a17b7c1e5504ae81" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44277 } } }
I20260619 09:05:04.760196 21797 sys_catalog.cc:458] T 00000000000000000000000000000000 P 291000932a274845a17b7c1e5504ae81 [sys.catalog]: This master's current role is: LEADER
W20260619 09:05:04.761247 21812 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 291000932a274845a17b7c1e5504ae81: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 09:05:04.761332 21812 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 09:05:04.761387 21800 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 09:05:04.761521 21800 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 09:05:04.763059 21800 catalog_manager.cc:1347] Generated new cluster ID: f30c4726dbc34a90a75c4e4163fcde2f
I20260619 09:05:04.763104 21800 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 09:05:04.764213 21795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:05:04.758609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.131:41541 (local address 127.0.0.1:44277)
0619 09:05:04.758769 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0619 09:05:04.758784 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 09:05:04.758866 (+ 82us) server_negotiation.cc:407] Connection header received
0619 09:05:04.758981 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:05:04.758988 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:05:04.759048 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:05:04.759213 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:05:04.759424 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:05:04.761130 (+ 1706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:05:04.762219 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:05:04.762739 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:05:04.762889 (+ 150us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:05:04.763531 (+ 642us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:05:04.763535 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:05:04.763536 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 09:05:04.763725 (+ 189us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:05:04.763760 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 09:05:04.763774 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:05:04.763859 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0619 09:05:04.763968 (+ 109us) server_negotiation.cc:299] Negotiation successful
0619 09:05:04.763995 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":71,"threads_started":1}
I20260619 09:05:04.765067 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 09:05:04.778429 21795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:05:04.766428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.131:60897 (local address 127.0.0.1:44277)
0619 09:05:04.766459 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0619 09:05:04.766461 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:05:04.775019 (+ 8558us) server_negotiation.cc:407] Connection header received
0619 09:05:04.775096 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:05:04.775097 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:05:04.775108 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:05:04.775169 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:05:04.775325 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:05:04.776058 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:05:04.777080 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:05:04.777392 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:05:04.777443 (+ 51us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:05:04.778166 (+ 723us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:05:04.778167 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:05:04.778168 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 09:05:04.778173 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:05:04.778189 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 09:05:04.778194 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:05:04.778244 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0619 09:05:04.778332 (+ 88us) server_negotiation.cc:299] Negotiation successful
0619 09:05:04.778346 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
I20260619 09:05:04.794368 21800 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 09:05:04.794929 21800 catalog_manager.cc:1504] Loading token signing keys...
I20260619 09:05:04.798986 21800 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 291000932a274845a17b7c1e5504ae81: Generated new TSK 0
I20260619 09:05:04.799175 21800 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 09:05:04.803747 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ez5NB/build/release/bin/kudu with pid 21723
W20260619 09:05:04.807682 21708 connection.cc:535] client connection to 127.0.0.1:44277 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44277): Connection reset by peer (error 104)
2026-06-19T09:05:04Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1223 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781859658134510-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 09:05:04 dist-test-slave-mtdr krb5kdc[21822](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 09:05:04 dist-test-slave-mtdr krb5kdc[21822](info): set up 2 sockets
Jun 19 09:05:04 dist-test-slave-mtdr krb5kdc[21822](info): commencing operation
krb5kdc: starting...
W20260619 09:05:05.742647 392 mini_kdc.cc:121] Time spent starting KDC: real 0.921s user 0.001s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781859658134510-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781859658134510-392-0/krb5kdc/test-user.keytab.
Jun 19 09:05:05 dist-test-slave-mtdr krb5kdc[21822](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859905, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T09:05:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T09:05:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781859658134510-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781859658134510-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 09:05:05.853847 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ez5NB/build/release/bin/kudu
/tmp/dist-test-task4ez5NB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781859658134510-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781859658134510-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781859658134510-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781859658134510-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37355
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781859658134510-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37355
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781859658134510-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781859658134510-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781859658134510-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 09:05:05.933753 21835 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 09:05:05.933955 21835 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 09:05:05.933985 21835 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 09:05:05.935217 21835 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 09:05:05.935263 21835 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 09:05:05.935284 21835 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 09:05:05.935300 21835 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 09:05:05.935322 21835 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 09:05:05.937109 21835 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37109
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781859658134510-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781859658134510-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37355
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781859658134510-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37355
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781859658134510-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21835
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781859658134510-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 09:05:05.937340 21835 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 09:05:05.937558 21835 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 09:05:05.942215 21843 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:05:05.942312 21835 server_base.cc:1034] running on GCE node
W20260619 09:05:05.942215 21841 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:05:05.942445 21840 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:05:05.942687 21835 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 09:05:05.942979 21835 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 09:05:05.944124 21835 hybrid_clock.cc:648] HybridClock initialized: now 1781859905944109 us; error 25 us; skew 500 ppm
Jun 19 09:05:05 dist-test-slave-mtdr krb5kdc[21822](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859905, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 09:05:05.946436 21835 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 09:05:05.946899 21835 webserver.cc:460] Webserver started at http://127.0.0.1:41839/ using document root <none> and password file <none>
I20260619 09:05:05.947111 21835 fs_manager.cc:359] Metadata directory not provided
I20260619 09:05:05.947160 21835 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 09:05:05.947278 21835 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 09:05:05.948328 21835 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781859658134510-392-0/minicluster-data/master-0/data/instance:
uuid: "7c7ebcef92904111a2ffed3b937931df"
format_stamp: "Formatted at 2026-06-19 09:05:05 on dist-test-slave-mtdr"
server_key: "51d3f7f3655cfeb2ea56f8ea1d5e3214"
server_key_iv: "d096eb391c1b2030cbd22dee749cd57e"
server_key_version: "encryptionkey@0"
I20260619 09:05:05.948653 21835 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781859658134510-392-0/minicluster-data/master-0/wal/instance:
uuid: "7c7ebcef92904111a2ffed3b937931df"
format_stamp: "Formatted at 2026-06-19 09:05:05 on dist-test-slave-mtdr"
server_key: "51d3f7f3655cfeb2ea56f8ea1d5e3214"
server_key_iv: "d096eb391c1b2030cbd22dee749cd57e"
server_key_version: "encryptionkey@0"
I20260619 09:05:05.949828 21835 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 09:05:05.950527 21850 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 09:05:05.950680 21835 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 09:05:05.950766 21835 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781859658134510-392-0/minicluster-data/master-0/data,/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781859658134510-392-0/minicluster-data/master-0/wal
uuid: "7c7ebcef92904111a2ffed3b937931df"
format_stamp: "Formatted at 2026-06-19 09:05:05 on dist-test-slave-mtdr"
server_key: "51d3f7f3655cfeb2ea56f8ea1d5e3214"
server_key_iv: "d096eb391c1b2030cbd22dee749cd57e"
server_key_version: "encryptionkey@0"
I20260619 09:05:05.950850 21835 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781859658134510-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781859658134510-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781859658134510-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 09:05:05.953982 21835 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 09:05:05.955956 21835 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 09:05:05.956077 21835 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 09:05:05.959942 21835 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37355
I20260619 09:05:05.959982 21902 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37355 every 8 connection(s)
I20260619 09:05:05.960263 21835 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781859658134510-392-0/minicluster-data/master-0/data/info.pb
I20260619 09:05:05.961198 21903 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 09:05:05.963406 21903 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7c7ebcef92904111a2ffed3b937931df: Bootstrap starting.
I20260619 09:05:05.963907 21903 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7c7ebcef92904111a2ffed3b937931df: Neither blocks nor log segments found. Creating new log.
I20260619 09:05:05.964154 21903 log.cc:824] T 00000000000000000000000000000000 P 7c7ebcef92904111a2ffed3b937931df: Log is configured to *not* fsync() on all Append() calls
I20260619 09:05:05.964746 21903 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7c7ebcef92904111a2ffed3b937931df: No bootstrap required, opened a new log
I20260619 09:05:05.966282 21903 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7c7ebcef92904111a2ffed3b937931df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c7ebcef92904111a2ffed3b937931df" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37355 } }
I20260619 09:05:05.966415 21903 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7c7ebcef92904111a2ffed3b937931df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 09:05:05.966444 21903 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7c7ebcef92904111a2ffed3b937931df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c7ebcef92904111a2ffed3b937931df, State: Initialized, Role: FOLLOWER
I20260619 09:05:05.966586 21903 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7c7ebcef92904111a2ffed3b937931df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c7ebcef92904111a2ffed3b937931df" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37355 } }
I20260619 09:05:05.966647 21903 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7c7ebcef92904111a2ffed3b937931df [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 09:05:05.966666 21903 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7c7ebcef92904111a2ffed3b937931df [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 09:05:05.966694 21903 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7c7ebcef92904111a2ffed3b937931df [term 0 FOLLOWER]: Advancing to term 1
I20260619 09:05:05.967216 21903 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7c7ebcef92904111a2ffed3b937931df [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c7ebcef92904111a2ffed3b937931df" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37355 } }
I20260619 09:05:05.967304 21903 leader_election.cc:302] T 00000000000000000000000000000000 P 7c7ebcef92904111a2ffed3b937931df [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7c7ebcef92904111a2ffed3b937931df; no voters:
I20260619 09:05:05.967447 21903 leader_election.cc:288] T 00000000000000000000000000000000 P 7c7ebcef92904111a2ffed3b937931df [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 09:05:05.967480 21906 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7c7ebcef92904111a2ffed3b937931df [term 1 FOLLOWER]: Leader election won for term 1
I20260619 09:05:05.967608 21906 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7c7ebcef92904111a2ffed3b937931df [term 1 LEADER]: Becoming Leader. State: Replica: 7c7ebcef92904111a2ffed3b937931df, State: Running, Role: LEADER
I20260619 09:05:05.967676 21903 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7c7ebcef92904111a2ffed3b937931df [sys.catalog]: configured and running, proceeding with master startup.
I20260619 09:05:05.967694 21906 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7c7ebcef92904111a2ffed3b937931df [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c7ebcef92904111a2ffed3b937931df" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37355 } }
I20260619 09:05:05.968127 21907 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c7ebcef92904111a2ffed3b937931df [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7c7ebcef92904111a2ffed3b937931df" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c7ebcef92904111a2ffed3b937931df" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37355 } } }
I20260619 09:05:05.968215 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ez5NB/build/release/bin/kudu as pid 21835
I20260619 09:05:05.968253 21907 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c7ebcef92904111a2ffed3b937931df [sys.catalog]: This master's current role is: LEADER
I20260619 09:05:05.968318 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781859658134510-392-0/minicluster-data/master-0/wal/instance
I20260619 09:05:05.968509 392 external_mini_cluster.cc:1428] Setting key 7bf9ddd94f76d498c07cd2c03774183e
I20260619 09:05:05.968011 21908 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c7ebcef92904111a2ffed3b937931df [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7c7ebcef92904111a2ffed3b937931df. Latest consensus state: current_term: 1 leader_uuid: "7c7ebcef92904111a2ffed3b937931df" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c7ebcef92904111a2ffed3b937931df" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37355 } } }
I20260619 09:05:05.968757 21908 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c7ebcef92904111a2ffed3b937931df [sys.catalog]: This master's current role is: LEADER
I20260619 09:05:05.968884 21915 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 09:05:05.969448 21915 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 09:05:05.970937 21915 catalog_manager.cc:1347] Generated new cluster ID: 442b7e7f5f884494a77730274e5afada
I20260619 09:05:05.970989 21915 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 09:05:05.971007 21924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:05:05.969112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.131:36523 (local address 127.0.0.1:37355)
0619 09:05:05.969250 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0619 09:05:05.969258 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 09:05:05.969288 (+ 30us) server_negotiation.cc:407] Connection header received
0619 09:05:05.969375 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:05:05.969380 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:05:05.969402 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:05:05.969523 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:05:05.970552 (+ 1029us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:05:05.970556 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:05:05.970556 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 09:05:05.970560 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:05:05.970603 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 09:05:05.970618 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:05:05.970674 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0619 09:05:05.970770 (+ 96us) server_negotiation.cc:299] Negotiation successful
0619 09:05:05.970832 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90,"thread_start_us":56,"threads_started":1}
I20260619 09:05:05.971781 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 09:05:05.976029 21924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:05:05.975050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.131:53793 (local address 127.0.0.1:37355)
0619 09:05:05.975084 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0619 09:05:05.975086 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:05:05.975227 (+ 141us) server_negotiation.cc:407] Connection header received
0619 09:05:05.975277 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:05:05.975279 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:05:05.975284 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:05:05.975327 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:05:05.975833 (+ 506us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:05:05.975835 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:05:05.975835 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 09:05:05.975839 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:05:05.975852 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 09:05:05.975855 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:05:05.975900 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0619 09:05:05.975954 (+ 54us) server_negotiation.cc:299] Negotiation successful
0619 09:05:05.975967 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260619 09:05:05.978390 21915 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 09:05:05.978533 21915 catalog_manager.cc:1504] Loading token signing keys...
I20260619 09:05:05.984655 21915 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7c7ebcef92904111a2ffed3b937931df: Generated new TSK 0
I20260619 09:05:05.984807 21915 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 09:05:05.986444 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ez5NB/build/release/bin/kudu with pid 21835
2026-06-19T09:05:05Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1182 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781859658134510-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 09:05:06 dist-test-slave-mtdr krb5kdc[21934](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 09:05:06 dist-test-slave-mtdr krb5kdc[21934](info): set up 2 sockets
Jun 19 09:05:06 dist-test-slave-mtdr krb5kdc[21934](info): commencing operation
krb5kdc: starting...
W20260619 09:05:06.938702 392 mini_kdc.cc:121] Time spent starting KDC: real 0.935s user 0.001s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781859658134510-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781859658134510-392-0/krb5kdc/test-user.keytab.
Jun 19 09:05:07 dist-test-slave-mtdr krb5kdc[21934](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859907, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T09:05:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T09:05:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781859658134510-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781859658134510-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 09:05:07.049568 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ez5NB/build/release/bin/kudu
/tmp/dist-test-task4ez5NB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781859658134510-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781859658134510-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781859658134510-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781859658134510-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42343
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781859658134510-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42343
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781859658134510-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781859658134510-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781859658134510-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 09:05:07.126888 21947 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 09:05:07.127099 21947 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 09:05:07.127123 21947 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 09:05:07.128250 21947 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 09:05:07.128290 21947 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 09:05:07.128304 21947 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 09:05:07.128314 21947 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 09:05:07.128324 21947 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 09:05:07.129746 21947 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781859658134510-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781859658134510-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42343
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781859658134510-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42343
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781859658134510-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21947
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781859658134510-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 09:05:07.129961 21947 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 09:05:07.130182 21947 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 09:05:07.134795 21952 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:05:07.134850 21955 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:05:07.134989 21947 server_base.cc:1034] running on GCE node
W20260619 09:05:07.134860 21953 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:05:07.135282 21947 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 09:05:07.135567 21947 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 09:05:07.136735 21947 hybrid_clock.cc:648] HybridClock initialized: now 1781859907136716 us; error 33 us; skew 500 ppm
Jun 19 09:05:07 dist-test-slave-mtdr krb5kdc[21934](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859907, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 09:05:07.139243 21947 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 09:05:07.139657 21947 webserver.cc:460] Webserver started at http://127.0.98.62:44465/ using document root <none> and password file <none>
I20260619 09:05:07.139863 21947 fs_manager.cc:359] Metadata directory not provided
I20260619 09:05:07.139910 21947 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 09:05:07.140023 21947 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 09:05:07.141016 21947 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781859658134510-392-0/minicluster-data/master-0/data/instance:
uuid: "2ad8074c6051433787a0cbf206f1dfaf"
format_stamp: "Formatted at 2026-06-19 09:05:07 on dist-test-slave-mtdr"
server_key: "24002fb146f03731b968c5a7ae7fe451"
server_key_iv: "d70c84d1c580cb467ef6ce49303eebca"
server_key_version: "encryptionkey@0"
I20260619 09:05:07.141350 21947 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781859658134510-392-0/minicluster-data/master-0/wal/instance:
uuid: "2ad8074c6051433787a0cbf206f1dfaf"
format_stamp: "Formatted at 2026-06-19 09:05:07 on dist-test-slave-mtdr"
server_key: "24002fb146f03731b968c5a7ae7fe451"
server_key_iv: "d70c84d1c580cb467ef6ce49303eebca"
server_key_version: "encryptionkey@0"
I20260619 09:05:07.142552 21947 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260619 09:05:07.143245 21962 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 09:05:07.143390 21947 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260619 09:05:07.143468 21947 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781859658134510-392-0/minicluster-data/master-0/data,/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781859658134510-392-0/minicluster-data/master-0/wal
uuid: "2ad8074c6051433787a0cbf206f1dfaf"
format_stamp: "Formatted at 2026-06-19 09:05:07 on dist-test-slave-mtdr"
server_key: "24002fb146f03731b968c5a7ae7fe451"
server_key_iv: "d70c84d1c580cb467ef6ce49303eebca"
server_key_version: "encryptionkey@0"
I20260619 09:05:07.143532 21947 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781859658134510-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781859658134510-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781859658134510-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 09:05:07.157243 21947 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 09:05:07.159171 21947 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 09:05:07.159309 21947 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 09:05:07.162824 21947 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42343
I20260619 09:05:07.162887 22014 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42343 every 8 connection(s)
I20260619 09:05:07.163178 21947 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781859658134510-392-0/minicluster-data/master-0/data/info.pb
I20260619 09:05:07.163832 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ez5NB/build/release/bin/kudu as pid 21947
I20260619 09:05:07.163946 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781859658134510-392-0/minicluster-data/master-0/wal/instance
I20260619 09:05:07.164121 22015 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 09:05:07.164175 392 external_mini_cluster.cc:1428] Setting key 0e2a059b6cda1d1b9342ef8d8455ce7b
I20260619 09:05:07.167153 22015 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ad8074c6051433787a0cbf206f1dfaf: Bootstrap starting.
I20260619 09:05:07.167752 22015 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2ad8074c6051433787a0cbf206f1dfaf: Neither blocks nor log segments found. Creating new log.
I20260619 09:05:07.168043 22015 log.cc:824] T 00000000000000000000000000000000 P 2ad8074c6051433787a0cbf206f1dfaf: Log is configured to *not* fsync() on all Append() calls
I20260619 09:05:07.168730 22015 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ad8074c6051433787a0cbf206f1dfaf: No bootstrap required, opened a new log
Jun 19 09:05:07 dist-test-slave-mtdr krb5kdc[21934](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859907, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 09:05:07.170989 22015 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2ad8074c6051433787a0cbf206f1dfaf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ad8074c6051433787a0cbf206f1dfaf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42343 } }
I20260619 09:05:07.171135 22015 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2ad8074c6051433787a0cbf206f1dfaf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 09:05:07.171170 22015 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2ad8074c6051433787a0cbf206f1dfaf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ad8074c6051433787a0cbf206f1dfaf, State: Initialized, Role: FOLLOWER
I20260619 09:05:07.171288 22015 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2ad8074c6051433787a0cbf206f1dfaf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ad8074c6051433787a0cbf206f1dfaf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42343 } }
I20260619 09:05:07.171345 22015 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2ad8074c6051433787a0cbf206f1dfaf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 09:05:07.171375 22015 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2ad8074c6051433787a0cbf206f1dfaf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 09:05:07.171419 22015 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2ad8074c6051433787a0cbf206f1dfaf [term 0 FOLLOWER]: Advancing to term 1
I20260619 09:05:07.171919 22015 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2ad8074c6051433787a0cbf206f1dfaf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ad8074c6051433787a0cbf206f1dfaf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42343 } }
I20260619 09:05:07.172008 22015 leader_election.cc:302] T 00000000000000000000000000000000 P 2ad8074c6051433787a0cbf206f1dfaf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2ad8074c6051433787a0cbf206f1dfaf; no voters:
I20260619 09:05:07.172165 22015 leader_election.cc:288] T 00000000000000000000000000000000 P 2ad8074c6051433787a0cbf206f1dfaf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 09:05:07.172204 22020 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2ad8074c6051433787a0cbf206f1dfaf [term 1 FOLLOWER]: Leader election won for term 1
I20260619 09:05:07.172302 22020 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2ad8074c6051433787a0cbf206f1dfaf [term 1 LEADER]: Becoming Leader. State: Replica: 2ad8074c6051433787a0cbf206f1dfaf, State: Running, Role: LEADER
I20260619 09:05:07.172372 22020 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2ad8074c6051433787a0cbf206f1dfaf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ad8074c6051433787a0cbf206f1dfaf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42343 } }
I20260619 09:05:07.172415 22015 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2ad8074c6051433787a0cbf206f1dfaf [sys.catalog]: configured and running, proceeding with master startup.
I20260619 09:05:07.172649 22020 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ad8074c6051433787a0cbf206f1dfaf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2ad8074c6051433787a0cbf206f1dfaf. Latest consensus state: current_term: 1 leader_uuid: "2ad8074c6051433787a0cbf206f1dfaf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ad8074c6051433787a0cbf206f1dfaf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42343 } } }
I20260619 09:05:07.172771 22020 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ad8074c6051433787a0cbf206f1dfaf [sys.catalog]: This master's current role is: LEADER
I20260619 09:05:07.173211 22026 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 09:05:07.173702 22026 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 09:05:07.174391 22021 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ad8074c6051433787a0cbf206f1dfaf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2ad8074c6051433787a0cbf206f1dfaf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ad8074c6051433787a0cbf206f1dfaf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42343 } } }
I20260619 09:05:07.174532 22021 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ad8074c6051433787a0cbf206f1dfaf [sys.catalog]: This master's current role is: LEADER
I20260619 09:05:07.175539 22018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:05:07.164674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34267 (local address 127.0.98.62:42343)
0619 09:05:07.164840 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0619 09:05:07.164847 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 09:05:07.164871 (+ 24us) server_negotiation.cc:407] Connection header received
0619 09:05:07.165463 (+ 592us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:05:07.165471 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:05:07.165519 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:05:07.165637 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:05:07.166440 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:05:07.167458 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:05:07.168442 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:05:07.168905 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:05:07.168988 (+ 83us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:05:07.171526 (+ 2538us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:05:07.171529 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:05:07.171530 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:05:07.171540 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:05:07.173391 (+ 1851us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:05:07.173840 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:05:07.173843 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:05:07.173843 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:05:07.173938 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:05:07.174384 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:05:07.174385 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:05:07.174386 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:05:07.174701 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:05:07.174896 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0619 09:05:07.175248 (+ 352us) server_negotiation.cc:299] Negotiation successful
0619 09:05:07.175347 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":63,"threads_started":1}
I20260619 09:05:07.175961 22026 catalog_manager.cc:1347] Generated new cluster ID: 9ba4fb49a54a465696d7efdaeb3b2e3d
I20260619 09:05:07.176054 22026 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 09:05:07.176517 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 09:05:07.194197 22018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:05:07.187135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35545 (local address 127.0.98.62:42343)
0619 09:05:07.187183 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0619 09:05:07.187185 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:05:07.187290 (+ 105us) server_negotiation.cc:407] Connection header received
0619 09:05:07.187341 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:05:07.187342 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:05:07.187352 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:05:07.187408 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:05:07.188033 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:05:07.188806 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:05:07.189745 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:05:07.190098 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:05:07.190167 (+ 69us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:05:07.191534 (+ 1367us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:05:07.191536 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:05:07.191536 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:05:07.191544 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:05:07.193046 (+ 1502us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:05:07.193465 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:05:07.193466 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:05:07.193466 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:05:07.193534 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:05:07.193722 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:05:07.193723 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:05:07.193723 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:05:07.193851 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:05:07.193949 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0619 09:05:07.194042 (+ 93us) server_negotiation.cc:299] Negotiation successful
0619 09:05:07.194111 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260619 09:05:07.199399 22026 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 09:05:07.199913 22026 catalog_manager.cc:1504] Loading token signing keys...
I20260619 09:05:07.202435 22026 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2ad8074c6051433787a0cbf206f1dfaf: Generated new TSK 0
I20260619 09:05:07.202593 22026 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 09:05:07.205116 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ez5NB/build/release/bin/kudu with pid 21947
W20260619 09:05:07.208961 21932 connection.cc:535] client connection to 127.0.98.62:42343 recv error: Network error: failed to read from TLS socket (remote: 127.0.98.62:42343): Connection reset by peer (error 104)
2026-06-19T09:05:07Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1218 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781859658134510-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 09:05:07 dist-test-slave-mtdr krb5kdc[22046](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 09:05:07 dist-test-slave-mtdr krb5kdc[22046](info): set up 2 sockets
Jun 19 09:05:07 dist-test-slave-mtdr krb5kdc[22046](info): commencing operation
krb5kdc: starting...
W20260619 09:05:08.158641 392 mini_kdc.cc:121] Time spent starting KDC: real 0.937s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781859658134510-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781859658134510-392-0/krb5kdc/test-user.keytab.
Jun 19 09:05:08 dist-test-slave-mtdr krb5kdc[22046](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859908, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T09:05:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T09:05:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781859658134510-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781859658134510-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 09:05:08.273707 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ez5NB/build/release/bin/kudu
/tmp/dist-test-task4ez5NB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781859658134510-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781859658134510-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781859658134510-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781859658134510-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46221
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781859658134510-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46803
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46221
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781859658134510-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781859658134510-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781859658134510-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 09:05:08.351733 22059 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 09:05:08.351948 22059 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 09:05:08.351969 22059 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 09:05:08.353052 22059 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 09:05:08.353086 22059 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 09:05:08.353099 22059 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 09:05:08.353111 22059 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 09:05:08.353121 22059 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 09:05:08.354512 22059 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46803
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781859658134510-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781859658134510-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46221
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781859658134510-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46221
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781859658134510-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22059
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781859658134510-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 09:05:08.354699 22059 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 09:05:08.354907 22059 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 09:05:08.359653 22065 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:05:08.359663 22067 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:05:08.359652 22064 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:05:08.359869 22059 server_base.cc:1034] running on GCE node
I20260619 09:05:08.360083 22059 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 09:05:08.360350 22059 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 09:05:08.361493 22059 hybrid_clock.cc:648] HybridClock initialized: now 1781859908361476 us; error 28 us; skew 500 ppm
Jun 19 09:05:08 dist-test-slave-mtdr krb5kdc[22046](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859908, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 09:05:08.368626 22059 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 09:05:08.369064 22059 webserver.cc:460] Webserver started at http://127.0.98.62:42819/ using document root <none> and password file <none>
I20260619 09:05:08.369266 22059 fs_manager.cc:359] Metadata directory not provided
I20260619 09:05:08.369314 22059 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 09:05:08.369433 22059 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 09:05:08.370467 22059 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781859658134510-392-0/minicluster-data/master-0/data/instance:
uuid: "ce0ace918be349d586503b6c49dedb23"
format_stamp: "Formatted at 2026-06-19 09:05:08 on dist-test-slave-mtdr"
server_key: "190b8b45d7af8e10edf6f26871bb1dfe"
server_key_iv: "6356cac627ff13456b68a45a903937a0"
server_key_version: "encryptionkey@0"
I20260619 09:05:08.370783 22059 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781859658134510-392-0/minicluster-data/master-0/wal/instance:
uuid: "ce0ace918be349d586503b6c49dedb23"
format_stamp: "Formatted at 2026-06-19 09:05:08 on dist-test-slave-mtdr"
server_key: "190b8b45d7af8e10edf6f26871bb1dfe"
server_key_iv: "6356cac627ff13456b68a45a903937a0"
server_key_version: "encryptionkey@0"
I20260619 09:05:08.372056 22059 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260619 09:05:08.372761 22074 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 09:05:08.372915 22059 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 09:05:08.373003 22059 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781859658134510-392-0/minicluster-data/master-0/data,/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781859658134510-392-0/minicluster-data/master-0/wal
uuid: "ce0ace918be349d586503b6c49dedb23"
format_stamp: "Formatted at 2026-06-19 09:05:08 on dist-test-slave-mtdr"
server_key: "190b8b45d7af8e10edf6f26871bb1dfe"
server_key_iv: "6356cac627ff13456b68a45a903937a0"
server_key_version: "encryptionkey@0"
I20260619 09:05:08.373061 22059 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781859658134510-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781859658134510-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781859658134510-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 09:05:08.385682 22059 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 09:05:08.387725 22059 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 09:05:08.387861 22059 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 09:05:08.391649 22059 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46221
I20260619 09:05:08.391682 22126 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46221 every 8 connection(s)
I20260619 09:05:08.392025 22059 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781859658134510-392-0/minicluster-data/master-0/data/info.pb
I20260619 09:05:08.392948 22127 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 09:05:08.395188 22127 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ce0ace918be349d586503b6c49dedb23: Bootstrap starting.
I20260619 09:05:08.395721 22127 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ce0ace918be349d586503b6c49dedb23: Neither blocks nor log segments found. Creating new log.
I20260619 09:05:08.395980 22127 log.cc:824] T 00000000000000000000000000000000 P ce0ace918be349d586503b6c49dedb23: Log is configured to *not* fsync() on all Append() calls
I20260619 09:05:08.396612 22127 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ce0ace918be349d586503b6c49dedb23: No bootstrap required, opened a new log
I20260619 09:05:08.398146 22127 raft_consensus.cc:348] T 00000000000000000000000000000000 P ce0ace918be349d586503b6c49dedb23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce0ace918be349d586503b6c49dedb23" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46221 } }
I20260619 09:05:08.398298 22127 raft_consensus.cc:374] T 00000000000000000000000000000000 P ce0ace918be349d586503b6c49dedb23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 09:05:08.398320 22127 raft_consensus.cc:729] T 00000000000000000000000000000000 P ce0ace918be349d586503b6c49dedb23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce0ace918be349d586503b6c49dedb23, State: Initialized, Role: FOLLOWER
I20260619 09:05:08.398306 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ez5NB/build/release/bin/kudu as pid 22059
I20260619 09:05:08.398409 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781859658134510-392-0/minicluster-data/master-0/wal/instance
I20260619 09:05:08.398445 22127 consensus_queue.cc:260] T 00000000000000000000000000000000 P ce0ace918be349d586503b6c49dedb23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce0ace918be349d586503b6c49dedb23" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46221 } }
I20260619 09:05:08.398500 22127 raft_consensus.cc:388] T 00000000000000000000000000000000 P ce0ace918be349d586503b6c49dedb23 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 09:05:08.398525 22127 raft_consensus.cc:482] T 00000000000000000000000000000000 P ce0ace918be349d586503b6c49dedb23 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 09:05:08.398572 22127 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ce0ace918be349d586503b6c49dedb23 [term 0 FOLLOWER]: Advancing to term 1
I20260619 09:05:08.398609 392 external_mini_cluster.cc:1428] Setting key 3321a16ffd85a43ac7dcd8425b9137d4
I20260619 09:05:08.399216 22127 raft_consensus.cc:504] T 00000000000000000000000000000000 P ce0ace918be349d586503b6c49dedb23 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce0ace918be349d586503b6c49dedb23" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46221 } }
I20260619 09:05:08.399326 22127 leader_election.cc:302] T 00000000000000000000000000000000 P ce0ace918be349d586503b6c49dedb23 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ce0ace918be349d586503b6c49dedb23; no voters:
I20260619 09:05:08.399463 22127 leader_election.cc:288] T 00000000000000000000000000000000 P ce0ace918be349d586503b6c49dedb23 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 09:05:08.399489 22132 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ce0ace918be349d586503b6c49dedb23 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 09:05:08.399645 22132 raft_consensus.cc:686] T 00000000000000000000000000000000 P ce0ace918be349d586503b6c49dedb23 [term 1 LEADER]: Becoming Leader. State: Replica: ce0ace918be349d586503b6c49dedb23, State: Running, Role: LEADER
I20260619 09:05:08.399696 22127 sys_catalog.cc:564] T 00000000000000000000000000000000 P ce0ace918be349d586503b6c49dedb23 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 09:05:08.399781 22132 consensus_queue.cc:237] T 00000000000000000000000000000000 P ce0ace918be349d586503b6c49dedb23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce0ace918be349d586503b6c49dedb23" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46221 } }
I20260619 09:05:08.400141 22133 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce0ace918be349d586503b6c49dedb23 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ce0ace918be349d586503b6c49dedb23" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce0ace918be349d586503b6c49dedb23" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46221 } } }
I20260619 09:05:08.400223 22133 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce0ace918be349d586503b6c49dedb23 [sys.catalog]: This master's current role is: LEADER
I20260619 09:05:08.400367 22134 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce0ace918be349d586503b6c49dedb23 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ce0ace918be349d586503b6c49dedb23. Latest consensus state: current_term: 1 leader_uuid: "ce0ace918be349d586503b6c49dedb23" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce0ace918be349d586503b6c49dedb23" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46221 } } }
I20260619 09:05:08.400465 22134 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce0ace918be349d586503b6c49dedb23 [sys.catalog]: This master's current role is: LEADER
W20260619 09:05:08.401768 22148 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ce0ace918be349d586503b6c49dedb23: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 09:05:08.401875 22148 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 09:05:08.401944 22138 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 09:05:08.402112 22138 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 09:05:08.403939 22138 catalog_manager.cc:1347] Generated new cluster ID: 200351a5c5654c1481d6eefd985dc901
I20260619 09:05:08.403993 22138 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 09:05:08.409956 22138 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 09:05:08.410476 22131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:05:08.399102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33189 (local address 127.0.98.62:46221)
0619 09:05:08.399299 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0619 09:05:08.399308 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 09:05:08.399343 (+ 35us) server_negotiation.cc:407] Connection header received
0619 09:05:08.399459 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:05:08.399467 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:05:08.399527 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:05:08.399945 (+ 418us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:05:08.400033 (+ 88us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:05:08.401733 (+ 1700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:05:08.402832 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:05:08.403326 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:05:08.403433 (+ 107us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:05:08.409735 (+ 6302us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:05:08.409739 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:05:08.409740 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 09:05:08.409960 (+ 220us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:05:08.410003 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 09:05:08.410019 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:05:08.410109 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0619 09:05:08.410228 (+ 119us) server_negotiation.cc:299] Negotiation successful
0619 09:05:08.410257 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":92,"threads_started":1}
I20260619 09:05:08.410602 22138 catalog_manager.cc:1504] Loading token signing keys...
I20260619 09:05:08.411444 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 09:05:08.414921 22138 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ce0ace918be349d586503b6c49dedb23: Generated new TSK 0
I20260619 09:05:08.415082 22138 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 09:05:08.416257 22131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:05:08.412888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37443 (local address 127.0.98.62:46221)
0619 09:05:08.412925 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0619 09:05:08.412926 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 09:05:08.413016 (+ 90us) server_negotiation.cc:407] Connection header received
0619 09:05:08.413063 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:05:08.413063 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:05:08.413072 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:05:08.413133 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:05:08.413259 (+ 126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:05:08.414002 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:05:08.415097 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:05:08.415372 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:05:08.415434 (+ 62us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:05:08.416012 (+ 578us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:05:08.416013 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:05:08.416013 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 09:05:08.416017 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:05:08.416029 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 09:05:08.416033 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:05:08.416081 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0619 09:05:08.416170 (+ 89us) server_negotiation.cc:299] Negotiation successful
0619 09:05:08.416184 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260619 09:05:08.418036 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ez5NB/build/release/bin/kudu with pid 22059
W20260619 09:05:08.421564 22044 connection.cc:535] client connection to 127.0.98.62:46221 recv error: Network error: failed to read from TLS socket (remote: 127.0.98.62:46221): Connection reset by peer (error 104)
2026-06-19T09:05:08Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1212 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781859658134510-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 09:05:08 dist-test-slave-mtdr krb5kdc[22158](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 09:05:08 dist-test-slave-mtdr krb5kdc[22158](info): set up 2 sockets
Jun 19 09:05:08 dist-test-slave-mtdr krb5kdc[22158](info): commencing operation
krb5kdc: starting...
W20260619 09:05:09.367347 392 mini_kdc.cc:121] Time spent starting KDC: real 0.932s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781859658134510-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781859658134510-392-0/krb5kdc/test-user.keytab.
Jun 19 09:05:09 dist-test-slave-mtdr krb5kdc[22158](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859909, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T09:05:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T09:05:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781859658134510-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781859658134510-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 09:05:09.485884 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ez5NB/build/release/bin/kudu
/tmp/dist-test-task4ez5NB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781859658134510-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781859658134510-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781859658134510-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781859658134510-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33615
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781859658134510-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33615
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781859658134510-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781859658134510-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781859658134510-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 09:05:09.566159 22171 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 09:05:09.566385 22171 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 09:05:09.566407 22171 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 09:05:09.567624 22171 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 09:05:09.567669 22171 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 09:05:09.567682 22171 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 09:05:09.567693 22171 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 09:05:09.567704 22171 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 09:05:09.569428 22171 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781859658134510-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781859658134510-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33615
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781859658134510-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33615
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781859658134510-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22171
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781859658134510-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 09:05:09.569689 22171 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 09:05:09.569941 22171 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 09:05:09.574844 22176 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:05:09.575510 22179 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:05:09.575467 22177 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:05:09.576104 22171 server_base.cc:1034] running on GCE node
I20260619 09:05:09.576520 22171 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 09:05:09.576828 22171 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 09:05:09.578106 22171 hybrid_clock.cc:648] HybridClock initialized: now 1781859909578092 us; error 30 us; skew 500 ppm
Jun 19 09:05:09 dist-test-slave-mtdr krb5kdc[22158](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859909, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 09:05:09.581127 22171 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 09:05:09.581650 22171 webserver.cc:460] Webserver started at http://127.0.98.62:35415/ using document root <none> and password file <none>
I20260619 09:05:09.581874 22171 fs_manager.cc:359] Metadata directory not provided
I20260619 09:05:09.581924 22171 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 09:05:09.582051 22171 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 09:05:09.583181 22171 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781859658134510-392-0/minicluster-data/master-0/data/instance:
uuid: "ccaf68188b184ec7b59acf1964cc40e4"
format_stamp: "Formatted at 2026-06-19 09:05:09 on dist-test-slave-mtdr"
server_key: "84ba11e4d2ab6d8542f328bee0b6c9b4"
server_key_iv: "c98f62428c1628a98726994f7fd272f2"
server_key_version: "encryptionkey@0"
I20260619 09:05:09.583524 22171 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781859658134510-392-0/minicluster-data/master-0/wal/instance:
uuid: "ccaf68188b184ec7b59acf1964cc40e4"
format_stamp: "Formatted at 2026-06-19 09:05:09 on dist-test-slave-mtdr"
server_key: "84ba11e4d2ab6d8542f328bee0b6c9b4"
server_key_iv: "c98f62428c1628a98726994f7fd272f2"
server_key_version: "encryptionkey@0"
I20260619 09:05:09.584872 22171 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 09:05:09.585942 22186 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 09:05:09.586207 22171 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 09:05:09.586304 22171 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781859658134510-392-0/minicluster-data/master-0/data,/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781859658134510-392-0/minicluster-data/master-0/wal
uuid: "ccaf68188b184ec7b59acf1964cc40e4"
format_stamp: "Formatted at 2026-06-19 09:05:09 on dist-test-slave-mtdr"
server_key: "84ba11e4d2ab6d8542f328bee0b6c9b4"
server_key_iv: "c98f62428c1628a98726994f7fd272f2"
server_key_version: "encryptionkey@0"
I20260619 09:05:09.586372 22171 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781859658134510-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781859658134510-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781859658134510-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 09:05:09.589805 22171 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 09:05:09.591917 22171 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 09:05:09.592048 22171 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 09:05:09.595894 22171 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33615
I20260619 09:05:09.595923 22238 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33615 every 8 connection(s)
I20260619 09:05:09.596241 22171 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781859658134510-392-0/minicluster-data/master-0/data/info.pb
I20260619 09:05:09.597179 22239 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 09:05:09.599455 22239 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ccaf68188b184ec7b59acf1964cc40e4: Bootstrap starting.
I20260619 09:05:09.599998 22239 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ccaf68188b184ec7b59acf1964cc40e4: Neither blocks nor log segments found. Creating new log.
I20260619 09:05:09.600289 22239 log.cc:824] T 00000000000000000000000000000000 P ccaf68188b184ec7b59acf1964cc40e4: Log is configured to *not* fsync() on all Append() calls
I20260619 09:05:09.600386 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ez5NB/build/release/bin/kudu as pid 22171
I20260619 09:05:09.600502 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781859658134510-392-0/minicluster-data/master-0/wal/instance
I20260619 09:05:09.600687 392 external_mini_cluster.cc:1428] Setting key ae903bcef88147af68d90294ca9ce39e
I20260619 09:05:09.601032 22239 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ccaf68188b184ec7b59acf1964cc40e4: No bootstrap required, opened a new log
I20260619 09:05:09.603049 22239 raft_consensus.cc:348] T 00000000000000000000000000000000 P ccaf68188b184ec7b59acf1964cc40e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccaf68188b184ec7b59acf1964cc40e4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33615 } }
I20260619 09:05:09.603183 22239 raft_consensus.cc:374] T 00000000000000000000000000000000 P ccaf68188b184ec7b59acf1964cc40e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 09:05:09.603245 22239 raft_consensus.cc:729] T 00000000000000000000000000000000 P ccaf68188b184ec7b59acf1964cc40e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ccaf68188b184ec7b59acf1964cc40e4, State: Initialized, Role: FOLLOWER
I20260619 09:05:09.603389 22239 consensus_queue.cc:260] T 00000000000000000000000000000000 P ccaf68188b184ec7b59acf1964cc40e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccaf68188b184ec7b59acf1964cc40e4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33615 } }
I20260619 09:05:09.603499 22239 raft_consensus.cc:388] T 00000000000000000000000000000000 P ccaf68188b184ec7b59acf1964cc40e4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 09:05:09.603540 22239 raft_consensus.cc:482] T 00000000000000000000000000000000 P ccaf68188b184ec7b59acf1964cc40e4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 09:05:09.603581 22239 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ccaf68188b184ec7b59acf1964cc40e4 [term 0 FOLLOWER]: Advancing to term 1
I20260619 09:05:09.603627 22243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:05:09.601145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60047 (local address 127.0.98.62:33615)
0619 09:05:09.601343 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0619 09:05:09.601353 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 09:05:09.601377 (+ 24us) server_negotiation.cc:407] Connection header received
0619 09:05:09.602200 (+ 823us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:05:09.602208 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:05:09.602237 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:05:09.602372 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:05:09.603049 (+ 677us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:05:09.603053 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:05:09.603053 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 09:05:09.603058 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:05:09.603094 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 09:05:09.603104 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:05:09.603157 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0619 09:05:09.603253 (+ 96us) server_negotiation.cc:299] Negotiation successful
0619 09:05:09.603295 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":96,"threads_started":1}
I20260619 09:05:09.604334 22239 raft_consensus.cc:504] T 00000000000000000000000000000000 P ccaf68188b184ec7b59acf1964cc40e4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccaf68188b184ec7b59acf1964cc40e4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33615 } }
I20260619 09:05:09.604450 22239 leader_election.cc:302] T 00000000000000000000000000000000 P ccaf68188b184ec7b59acf1964cc40e4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ccaf68188b184ec7b59acf1964cc40e4; no voters:
I20260619 09:05:09.604552 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 09:05:09.604604 22239 leader_election.cc:288] T 00000000000000000000000000000000 P ccaf68188b184ec7b59acf1964cc40e4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 09:05:09.604636 22244 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ccaf68188b184ec7b59acf1964cc40e4 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 09:05:09.604847 22244 raft_consensus.cc:686] T 00000000000000000000000000000000 P ccaf68188b184ec7b59acf1964cc40e4 [term 1 LEADER]: Becoming Leader. State: Replica: ccaf68188b184ec7b59acf1964cc40e4, State: Running, Role: LEADER
I20260619 09:05:09.605014 22244 consensus_queue.cc:237] T 00000000000000000000000000000000 P ccaf68188b184ec7b59acf1964cc40e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccaf68188b184ec7b59acf1964cc40e4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33615 } }
I20260619 09:05:09.604831 22239 sys_catalog.cc:564] T 00000000000000000000000000000000 P ccaf68188b184ec7b59acf1964cc40e4 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 09:05:09.605532 22245 sys_catalog.cc:455] T 00000000000000000000000000000000 P ccaf68188b184ec7b59acf1964cc40e4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ccaf68188b184ec7b59acf1964cc40e4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccaf68188b184ec7b59acf1964cc40e4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33615 } } }
I20260619 09:05:09.605609 22245 sys_catalog.cc:458] T 00000000000000000000000000000000 P ccaf68188b184ec7b59acf1964cc40e4 [sys.catalog]: This master's current role is: LEADER
I20260619 09:05:09.605806 22246 sys_catalog.cc:455] T 00000000000000000000000000000000 P ccaf68188b184ec7b59acf1964cc40e4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ccaf68188b184ec7b59acf1964cc40e4. Latest consensus state: current_term: 1 leader_uuid: "ccaf68188b184ec7b59acf1964cc40e4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccaf68188b184ec7b59acf1964cc40e4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33615 } } }
I20260619 09:05:09.606189 22246 sys_catalog.cc:458] T 00000000000000000000000000000000 P ccaf68188b184ec7b59acf1964cc40e4 [sys.catalog]: This master's current role is: LEADER
I20260619 09:05:09.606328 22248 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 09:05:09.606953 22248 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 09:05:09.608446 22243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:05:09.607107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40139 (local address 127.0.98.62:33615)
0619 09:05:09.607142 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0619 09:05:09.607144 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:05:09.607426 (+ 282us) server_negotiation.cc:407] Connection header received
0619 09:05:09.607506 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:05:09.607507 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:05:09.607512 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:05:09.607587 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:05:09.608260 (+ 673us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:05:09.608261 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:05:09.608262 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 09:05:09.608266 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:05:09.608279 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 09:05:09.608282 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:05:09.608328 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0619 09:05:09.608376 (+ 48us) server_negotiation.cc:299] Negotiation successful
0619 09:05:09.608388 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260619 09:05:09.608916 22248 catalog_manager.cc:1347] Generated new cluster ID: abcf26bfbbbd4cedb6e152259c2fd95b
I20260619 09:05:09.608978 22248 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 09:05:09.618599 22248 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 09:05:09.618772 22248 catalog_manager.cc:1504] Loading token signing keys...
I20260619 09:05:09.625223 22248 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ccaf68188b184ec7b59acf1964cc40e4: Generated new TSK 0
I20260619 09:05:09.625392 22248 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 09:05:09.630892 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ez5NB/build/release/bin/kudu with pid 22171
2026-06-19T09:05:09Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1212 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781859658134510-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 09:05:09 dist-test-slave-mtdr krb5kdc[22270](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 09:05:09 dist-test-slave-mtdr krb5kdc[22270](info): set up 2 sockets
Jun 19 09:05:09 dist-test-slave-mtdr krb5kdc[22270](info): commencing operation
krb5kdc: starting...
W20260619 09:05:10.586632 392 mini_kdc.cc:121] Time spent starting KDC: real 0.939s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781859658134510-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781859658134510-392-0/krb5kdc/test-user.keytab.
Jun 19 09:05:10 dist-test-slave-mtdr krb5kdc[22270](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859910, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T09:05:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T09:05:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781859658134510-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781859658134510-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 09:05:10.695603 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ez5NB/build/release/bin/kudu
/tmp/dist-test-task4ez5NB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781859658134510-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781859658134510-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781859658134510-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781859658134510-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36727
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781859658134510-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41577
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36727
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781859658134510-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781859658134510-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781859658134510-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 09:05:10.774753 22283 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 09:05:10.775002 22283 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 09:05:10.775027 22283 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 09:05:10.776165 22283 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 09:05:10.776206 22283 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 09:05:10.776218 22283 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 09:05:10.776229 22283 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 09:05:10.776242 22283 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 09:05:10.777688 22283 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41577
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781859658134510-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781859658134510-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36727
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781859658134510-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36727
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781859658134510-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22283
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781859658134510-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 09:05:10.777908 22283 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 09:05:10.778108 22283 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 09:05:10.782641 22289 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:05:10.782646 22291 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:05:10.782845 22283 server_base.cc:1034] running on GCE node
W20260619 09:05:10.782904 22288 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:05:10.783123 22283 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 09:05:10.783366 22283 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 09:05:10.784538 22283 hybrid_clock.cc:648] HybridClock initialized: now 1781859910784525 us; error 27 us; skew 500 ppm
Jun 19 09:05:10 dist-test-slave-mtdr krb5kdc[22270](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859910, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 09:05:10.787014 22283 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 09:05:10.787459 22283 webserver.cc:460] Webserver started at http://127.0.98.62:43257/ using document root <none> and password file <none>
I20260619 09:05:10.787650 22283 fs_manager.cc:359] Metadata directory not provided
I20260619 09:05:10.787693 22283 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 09:05:10.787783 22283 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 09:05:10.788760 22283 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781859658134510-392-0/minicluster-data/master-0/data/instance:
uuid: "48f1d72739e64d79abb7cef377b8f56a"
format_stamp: "Formatted at 2026-06-19 09:05:10 on dist-test-slave-mtdr"
server_key: "7409df13effe442d6e8ae2afb64f2150"
server_key_iv: "34d918077162a4215ccb8d617794df70"
server_key_version: "encryptionkey@0"
I20260619 09:05:10.789060 22283 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781859658134510-392-0/minicluster-data/master-0/wal/instance:
uuid: "48f1d72739e64d79abb7cef377b8f56a"
format_stamp: "Formatted at 2026-06-19 09:05:10 on dist-test-slave-mtdr"
server_key: "7409df13effe442d6e8ae2afb64f2150"
server_key_iv: "34d918077162a4215ccb8d617794df70"
server_key_version: "encryptionkey@0"
I20260619 09:05:10.790359 22283 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 09:05:10.791123 22298 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 09:05:10.791273 22283 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260619 09:05:10.791344 22283 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781859658134510-392-0/minicluster-data/master-0/data,/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781859658134510-392-0/minicluster-data/master-0/wal
uuid: "48f1d72739e64d79abb7cef377b8f56a"
format_stamp: "Formatted at 2026-06-19 09:05:10 on dist-test-slave-mtdr"
server_key: "7409df13effe442d6e8ae2afb64f2150"
server_key_iv: "34d918077162a4215ccb8d617794df70"
server_key_version: "encryptionkey@0"
I20260619 09:05:10.791406 22283 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781859658134510-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781859658134510-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781859658134510-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 09:05:10.804044 22283 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 09:05:10.805823 22283 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 09:05:10.805930 22283 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 09:05:10.809474 22283 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36727
I20260619 09:05:10.809520 22350 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36727 every 8 connection(s)
I20260619 09:05:10.809857 22283 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781859658134510-392-0/minicluster-data/master-0/data/info.pb
I20260619 09:05:10.810667 22351 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 09:05:10.812759 22351 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 48f1d72739e64d79abb7cef377b8f56a: Bootstrap starting.
I20260619 09:05:10.813271 22351 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 48f1d72739e64d79abb7cef377b8f56a: Neither blocks nor log segments found. Creating new log.
I20260619 09:05:10.813513 22351 log.cc:824] T 00000000000000000000000000000000 P 48f1d72739e64d79abb7cef377b8f56a: Log is configured to *not* fsync() on all Append() calls
I20260619 09:05:10.814106 22351 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 48f1d72739e64d79abb7cef377b8f56a: No bootstrap required, opened a new log
I20260619 09:05:10.815683 22351 raft_consensus.cc:348] T 00000000000000000000000000000000 P 48f1d72739e64d79abb7cef377b8f56a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48f1d72739e64d79abb7cef377b8f56a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36727 } }
I20260619 09:05:10.815810 22351 raft_consensus.cc:374] T 00000000000000000000000000000000 P 48f1d72739e64d79abb7cef377b8f56a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 09:05:10.815831 22351 raft_consensus.cc:729] T 00000000000000000000000000000000 P 48f1d72739e64d79abb7cef377b8f56a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48f1d72739e64d79abb7cef377b8f56a, State: Initialized, Role: FOLLOWER
I20260619 09:05:10.815963 22351 consensus_queue.cc:260] T 00000000000000000000000000000000 P 48f1d72739e64d79abb7cef377b8f56a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48f1d72739e64d79abb7cef377b8f56a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36727 } }
I20260619 09:05:10.816022 22351 raft_consensus.cc:388] T 00000000000000000000000000000000 P 48f1d72739e64d79abb7cef377b8f56a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 09:05:10.816043 22351 raft_consensus.cc:482] T 00000000000000000000000000000000 P 48f1d72739e64d79abb7cef377b8f56a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 09:05:10.816087 22351 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 48f1d72739e64d79abb7cef377b8f56a [term 0 FOLLOWER]: Advancing to term 1
I20260619 09:05:10.816563 22351 raft_consensus.cc:504] T 00000000000000000000000000000000 P 48f1d72739e64d79abb7cef377b8f56a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48f1d72739e64d79abb7cef377b8f56a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36727 } }
I20260619 09:05:10.816641 22351 leader_election.cc:302] T 00000000000000000000000000000000 P 48f1d72739e64d79abb7cef377b8f56a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 48f1d72739e64d79abb7cef377b8f56a; no voters:
I20260619 09:05:10.816780 22351 leader_election.cc:288] T 00000000000000000000000000000000 P 48f1d72739e64d79abb7cef377b8f56a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 09:05:10.816833 22354 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 48f1d72739e64d79abb7cef377b8f56a [term 1 FOLLOWER]: Leader election won for term 1
I20260619 09:05:10.817005 22351 sys_catalog.cc:564] T 00000000000000000000000000000000 P 48f1d72739e64d79abb7cef377b8f56a [sys.catalog]: configured and running, proceeding with master startup.
I20260619 09:05:10.817021 22354 raft_consensus.cc:686] T 00000000000000000000000000000000 P 48f1d72739e64d79abb7cef377b8f56a [term 1 LEADER]: Becoming Leader. State: Replica: 48f1d72739e64d79abb7cef377b8f56a, State: Running, Role: LEADER
I20260619 09:05:10.817173 22354 consensus_queue.cc:237] T 00000000000000000000000000000000 P 48f1d72739e64d79abb7cef377b8f56a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48f1d72739e64d79abb7cef377b8f56a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36727 } }
I20260619 09:05:10.817736 22356 sys_catalog.cc:455] T 00000000000000000000000000000000 P 48f1d72739e64d79abb7cef377b8f56a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 48f1d72739e64d79abb7cef377b8f56a. Latest consensus state: current_term: 1 leader_uuid: "48f1d72739e64d79abb7cef377b8f56a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48f1d72739e64d79abb7cef377b8f56a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36727 } } }
I20260619 09:05:10.817837 22356 sys_catalog.cc:458] T 00000000000000000000000000000000 P 48f1d72739e64d79abb7cef377b8f56a [sys.catalog]: This master's current role is: LEADER
I20260619 09:05:10.817751 22355 sys_catalog.cc:455] T 00000000000000000000000000000000 P 48f1d72739e64d79abb7cef377b8f56a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "48f1d72739e64d79abb7cef377b8f56a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48f1d72739e64d79abb7cef377b8f56a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36727 } } }
I20260619 09:05:10.818078 22355 sys_catalog.cc:458] T 00000000000000000000000000000000 P 48f1d72739e64d79abb7cef377b8f56a [sys.catalog]: This master's current role is: LEADER
I20260619 09:05:10.818089 22363 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 09:05:10.818605 22363 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 09:05:10.819695 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ez5NB/build/release/bin/kudu as pid 22283
I20260619 09:05:10.819810 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781859658134510-392-0/minicluster-data/master-0/wal/instance
I20260619 09:05:10.820020 392 external_mini_cluster.cc:1428] Setting key 5e23f539c5d46e0744a0c8859c650b7a
I20260619 09:05:10.820236 22363 catalog_manager.cc:1347] Generated new cluster ID: a2acf878e84b42ceaa10d3cb33065939
I20260619 09:05:10.820286 22363 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 19 09:05:10 dist-test-slave-mtdr krb5kdc[22270](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859910, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 09:05:10.831326 22374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:05:10.820610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38047 (local address 127.0.98.62:36727)
0619 09:05:10.820767 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0619 09:05:10.820776 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 09:05:10.820806 (+ 30us) server_negotiation.cc:407] Connection header received
0619 09:05:10.820912 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:05:10.820920 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:05:10.820982 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:05:10.821117 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:05:10.821868 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:05:10.823312 (+ 1444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:05:10.824347 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:05:10.824987 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:05:10.825108 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:05:10.827830 (+ 2722us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:05:10.827835 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:05:10.827835 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:05:10.827848 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:05:10.829756 (+ 1908us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:05:10.830138 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:05:10.830141 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:05:10.830141 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:05:10.830241 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:05:10.830419 (+ 178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:05:10.830419 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:05:10.830420 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:05:10.830703 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:05:10.830851 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0619 09:05:10.830961 (+ 110us) server_negotiation.cc:299] Negotiation successful
0619 09:05:10.831090 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":83,"threads_started":1}
I20260619 09:05:10.832243 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 09:05:10.841825 22374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:05:10.833633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36775 (local address 127.0.98.62:36727)
0619 09:05:10.833671 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0619 09:05:10.833673 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:05:10.835007 (+ 1334us) server_negotiation.cc:407] Connection header received
0619 09:05:10.835058 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:05:10.835059 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:05:10.835070 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:05:10.835139 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:05:10.835733 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:05:10.836465 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:05:10.837503 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:05:10.837813 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:05:10.837881 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:05:10.839234 (+ 1353us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:05:10.839235 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:05:10.839236 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:05:10.839242 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:05:10.840738 (+ 1496us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:05:10.841102 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:05:10.841103 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:05:10.841103 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:05:10.841162 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:05:10.841342 (+ 180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:05:10.841343 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:05:10.841343 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:05:10.841454 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:05:10.841551 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0619 09:05:10.841665 (+ 114us) server_negotiation.cc:299] Negotiation successful
0619 09:05:10.841740 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260619 09:05:10.842519 22363 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 09:05:10.843225 22363 catalog_manager.cc:1504] Loading token signing keys...
I20260619 09:05:10.848222 22363 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 48f1d72739e64d79abb7cef377b8f56a: Generated new TSK 0
I20260619 09:05:10.848387 22363 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 09:05:10.856014 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ez5NB/build/release/bin/kudu with pid 22283
2026-06-19T09:05:10Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1225 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781859658134510-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 09:05:10 dist-test-slave-mtdr krb5kdc[22382](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 09:05:10 dist-test-slave-mtdr krb5kdc[22382](info): set up 2 sockets
Jun 19 09:05:10 dist-test-slave-mtdr krb5kdc[22382](info): commencing operation
krb5kdc: starting...
W20260619 09:05:11.815651 392 mini_kdc.cc:121] Time spent starting KDC: real 0.943s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781859658134510-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781859658134510-392-0/krb5kdc/test-user.keytab.
Jun 19 09:05:11 dist-test-slave-mtdr krb5kdc[22382](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859911, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T09:05:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T09:05:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781859658134510-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781859658134510-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 09:05:11.926231 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ez5NB/build/release/bin/kudu
/tmp/dist-test-task4ez5NB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781859658134510-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781859658134510-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781859658134510-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781859658134510-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38145
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781859658134510-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36401
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38145
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781859658134510-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781859658134510-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781859658134510-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 09:05:12.002719 22395 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 09:05:12.002949 22395 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 09:05:12.002985 22395 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 09:05:12.004107 22395 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 09:05:12.004148 22395 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 09:05:12.004175 22395 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 09:05:12.004190 22395 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 09:05:12.004207 22395 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 09:05:12.005630 22395 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36401
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781859658134510-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781859658134510-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38145
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781859658134510-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38145
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781859658134510-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22395
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781859658134510-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 09:05:12.005849 22395 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 09:05:12.006044 22395 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 09:05:12.010524 22400 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:05:12.010510 22401 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:05:12.010557 22395 server_base.cc:1034] running on GCE node
W20260619 09:05:12.010510 22403 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:05:12.010996 22395 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 09:05:12.011287 22395 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 09:05:12.012436 22395 hybrid_clock.cc:648] HybridClock initialized: now 1781859912012424 us; error 29 us; skew 500 ppm
Jun 19 09:05:12 dist-test-slave-mtdr krb5kdc[22382](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859912, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 09:05:12.014794 22395 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 09:05:12.015219 22395 webserver.cc:460] Webserver started at http://127.0.98.62:33407/ using document root <none> and password file <none>
I20260619 09:05:12.015426 22395 fs_manager.cc:359] Metadata directory not provided
I20260619 09:05:12.015475 22395 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 09:05:12.015574 22395 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 09:05:12.016590 22395 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781859658134510-392-0/minicluster-data/master-0/data/instance:
uuid: "0e958e40e65246e4ae577226d8873bab"
format_stamp: "Formatted at 2026-06-19 09:05:12 on dist-test-slave-mtdr"
server_key: "00a4377addd260e71eecd8922fe52037"
server_key_iv: "c4acc6be2dd761634de737104c143589"
server_key_version: "encryptionkey@0"
I20260619 09:05:12.016893 22395 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781859658134510-392-0/minicluster-data/master-0/wal/instance:
uuid: "0e958e40e65246e4ae577226d8873bab"
format_stamp: "Formatted at 2026-06-19 09:05:12 on dist-test-slave-mtdr"
server_key: "00a4377addd260e71eecd8922fe52037"
server_key_iv: "c4acc6be2dd761634de737104c143589"
server_key_version: "encryptionkey@0"
I20260619 09:05:12.018105 22395 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 09:05:12.018792 22410 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 09:05:12.018962 22395 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 09:05:12.019078 22395 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781859658134510-392-0/minicluster-data/master-0/data,/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781859658134510-392-0/minicluster-data/master-0/wal
uuid: "0e958e40e65246e4ae577226d8873bab"
format_stamp: "Formatted at 2026-06-19 09:05:12 on dist-test-slave-mtdr"
server_key: "00a4377addd260e71eecd8922fe52037"
server_key_iv: "c4acc6be2dd761634de737104c143589"
server_key_version: "encryptionkey@0"
I20260619 09:05:12.019138 22395 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781859658134510-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781859658134510-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781859658134510-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 09:05:12.028127 22395 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 09:05:12.030022 22395 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 09:05:12.030148 22395 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 09:05:12.033666 22395 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38145
I20260619 09:05:12.033704 22462 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38145 every 8 connection(s)
I20260619 09:05:12.033987 22395 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781859658134510-392-0/minicluster-data/master-0/data/info.pb
I20260619 09:05:12.034801 22463 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 09:05:12.037104 22463 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0e958e40e65246e4ae577226d8873bab: Bootstrap starting.
I20260619 09:05:12.037648 22463 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0e958e40e65246e4ae577226d8873bab: Neither blocks nor log segments found. Creating new log.
I20260619 09:05:12.037874 22463 log.cc:824] T 00000000000000000000000000000000 P 0e958e40e65246e4ae577226d8873bab: Log is configured to *not* fsync() on all Append() calls
I20260619 09:05:12.038493 22463 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0e958e40e65246e4ae577226d8873bab: No bootstrap required, opened a new log
I20260619 09:05:12.040145 22463 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0e958e40e65246e4ae577226d8873bab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e958e40e65246e4ae577226d8873bab" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38145 } }
I20260619 09:05:12.040235 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ez5NB/build/release/bin/kudu as pid 22395
I20260619 09:05:12.040287 22463 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0e958e40e65246e4ae577226d8873bab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 09:05:12.040313 22463 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0e958e40e65246e4ae577226d8873bab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e958e40e65246e4ae577226d8873bab, State: Initialized, Role: FOLLOWER
I20260619 09:05:12.040334 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781859658134510-392-0/minicluster-data/master-0/wal/instance
I20260619 09:05:12.040441 22463 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0e958e40e65246e4ae577226d8873bab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e958e40e65246e4ae577226d8873bab" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38145 } }
I20260619 09:05:12.040504 22463 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0e958e40e65246e4ae577226d8873bab [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 09:05:12.040539 22463 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0e958e40e65246e4ae577226d8873bab [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 09:05:12.040541 392 external_mini_cluster.cc:1428] Setting key 2a8e1d50f7f84acd34c6f2b805cf0a1d
I20260619 09:05:12.040589 22463 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0e958e40e65246e4ae577226d8873bab [term 0 FOLLOWER]: Advancing to term 1
I20260619 09:05:12.041610 22463 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0e958e40e65246e4ae577226d8873bab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e958e40e65246e4ae577226d8873bab" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38145 } }
I20260619 09:05:12.041718 22463 leader_election.cc:302] T 00000000000000000000000000000000 P 0e958e40e65246e4ae577226d8873bab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0e958e40e65246e4ae577226d8873bab; no voters:
I20260619 09:05:12.041887 22463 leader_election.cc:288] T 00000000000000000000000000000000 P 0e958e40e65246e4ae577226d8873bab [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 09:05:12.041917 22468 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0e958e40e65246e4ae577226d8873bab [term 1 FOLLOWER]: Leader election won for term 1
I20260619 09:05:12.042034 22468 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0e958e40e65246e4ae577226d8873bab [term 1 LEADER]: Becoming Leader. State: Replica: 0e958e40e65246e4ae577226d8873bab, State: Running, Role: LEADER
I20260619 09:05:12.042137 22463 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0e958e40e65246e4ae577226d8873bab [sys.catalog]: configured and running, proceeding with master startup.
I20260619 09:05:12.042173 22468 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0e958e40e65246e4ae577226d8873bab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e958e40e65246e4ae577226d8873bab" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38145 } }
I20260619 09:05:12.042569 22469 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e958e40e65246e4ae577226d8873bab [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0e958e40e65246e4ae577226d8873bab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e958e40e65246e4ae577226d8873bab" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38145 } } }
I20260619 09:05:12.042579 22470 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e958e40e65246e4ae577226d8873bab [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0e958e40e65246e4ae577226d8873bab. Latest consensus state: current_term: 1 leader_uuid: "0e958e40e65246e4ae577226d8873bab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e958e40e65246e4ae577226d8873bab" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38145 } } }
I20260619 09:05:12.042670 22469 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e958e40e65246e4ae577226d8873bab [sys.catalog]: This master's current role is: LEADER
I20260619 09:05:12.042687 22470 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e958e40e65246e4ae577226d8873bab [sys.catalog]: This master's current role is: LEADER
I20260619 09:05:12.043016 22476 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 09:05:12.043571 22476 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 09:05:12.045164 22476 catalog_manager.cc:1347] Generated new cluster ID: fb9bf743ea134f1db4731e50ff82aeec
I20260619 09:05:12.045215 22476 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 09:05:12.046401 22467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:05:12.041000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60167 (local address 127.0.98.62:38145)
0619 09:05:12.041236 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0619 09:05:12.041245 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 09:05:12.041284 (+ 39us) server_negotiation.cc:407] Connection header received
0619 09:05:12.041400 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:05:12.041407 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:05:12.041462 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:05:12.041613 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:05:12.041837 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:05:12.043181 (+ 1344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:05:12.044285 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:05:12.044840 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:05:12.044940 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:05:12.045623 (+ 683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:05:12.045626 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:05:12.045627 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 09:05:12.045839 (+ 212us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:05:12.045876 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 09:05:12.045890 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:05:12.045942 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0619 09:05:12.046129 (+ 187us) server_negotiation.cc:299] Negotiation successful
0619 09:05:12.046187 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":117,"threads_started":1}
I20260619 09:05:12.047300 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 09:05:12.054385 22467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:05:12.051086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48545 (local address 127.0.98.62:38145)
0619 09:05:12.051127 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0619 09:05:12.051129 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:05:12.051230 (+ 101us) server_negotiation.cc:407] Connection header received
0619 09:05:12.051277 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:05:12.051278 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:05:12.051288 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:05:12.051346 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:05:12.051464 (+ 118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:05:12.052181 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:05:12.053185 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:05:12.053474 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:05:12.053536 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:05:12.054151 (+ 615us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:05:12.054152 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:05:12.054152 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 09:05:12.054158 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:05:12.054173 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 09:05:12.054177 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:05:12.054231 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0619 09:05:12.054283 (+ 52us) server_negotiation.cc:299] Negotiation successful
0619 09:05:12.054308 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260619 09:05:12.066370 22476 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 09:05:12.066933 22476 catalog_manager.cc:1504] Loading token signing keys...
I20260619 09:05:12.070020 22476 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0e958e40e65246e4ae577226d8873bab: Generated new TSK 0
I20260619 09:05:12.070180 22476 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 09:05:12.073297 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ez5NB/build/release/bin/kudu with pid 22395
2026-06-19T09:05:12Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1217 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781859658134510-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 09:05:12 dist-test-slave-mtdr krb5kdc[22494](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 09:05:12 dist-test-slave-mtdr krb5kdc[22494](info): set up 2 sockets
Jun 19 09:05:12 dist-test-slave-mtdr krb5kdc[22494](info): commencing operation
krb5kdc: starting...
W20260619 09:05:13.038674 392 mini_kdc.cc:121] Time spent starting KDC: real 0.948s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781859658134510-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781859658134510-392-0/krb5kdc/test-user.keytab.
Jun 19 09:05:13 dist-test-slave-mtdr krb5kdc[22494](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859913, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T09:05:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T09:05:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781859658134510-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781859658134510-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 09:05:13.147581 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ez5NB/build/release/bin/kudu
/tmp/dist-test-task4ez5NB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781859658134510-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781859658134510-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781859658134510-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781859658134510-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41827
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781859658134510-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41827
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781859658134510-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781859658134510-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781859658134510-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 09:05:13.223634 22507 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 09:05:13.223819 22507 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 09:05:13.223839 22507 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 09:05:13.224905 22507 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 09:05:13.224941 22507 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 09:05:13.224954 22507 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 09:05:13.224964 22507 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 09:05:13.224974 22507 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 09:05:13.226328 22507 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781859658134510-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781859658134510-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41827
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781859658134510-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41827
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781859658134510-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22507
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781859658134510-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 09:05:13.226500 22507 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 09:05:13.226658 22507 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 09:05:13.231300 22512 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:05:13.231353 22513 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:05:13.231353 22515 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:05:13.231379 22507 server_base.cc:1034] running on GCE node
I20260619 09:05:13.231801 22507 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 09:05:13.232061 22507 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 09:05:13.233206 22507 hybrid_clock.cc:648] HybridClock initialized: now 1781859913233174 us; error 50 us; skew 500 ppm
Jun 19 09:05:13 dist-test-slave-mtdr krb5kdc[22494](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859913, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 09:05:13.235580 22507 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 09:05:13.235978 22507 webserver.cc:460] Webserver started at http://127.0.98.62:42521/ using document root <none> and password file <none>
I20260619 09:05:13.236145 22507 fs_manager.cc:359] Metadata directory not provided
I20260619 09:05:13.236181 22507 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 09:05:13.236274 22507 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 09:05:13.237224 22507 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781859658134510-392-0/minicluster-data/master-0/data/instance:
uuid: "0c027bf9265047459a0880a8e28508f3"
format_stamp: "Formatted at 2026-06-19 09:05:13 on dist-test-slave-mtdr"
server_key: "51b27b014da1428c3e9df0e1880528a6"
server_key_iv: "fdf7619ecd0ad5607520b3b686f3d87c"
server_key_version: "encryptionkey@0"
I20260619 09:05:13.237502 22507 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781859658134510-392-0/minicluster-data/master-0/wal/instance:
uuid: "0c027bf9265047459a0880a8e28508f3"
format_stamp: "Formatted at 2026-06-19 09:05:13 on dist-test-slave-mtdr"
server_key: "51b27b014da1428c3e9df0e1880528a6"
server_key_iv: "fdf7619ecd0ad5607520b3b686f3d87c"
server_key_version: "encryptionkey@0"
I20260619 09:05:13.238639 22507 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260619 09:05:13.239405 22522 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 09:05:13.239591 22507 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260619 09:05:13.239717 22507 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781859658134510-392-0/minicluster-data/master-0/data,/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781859658134510-392-0/minicluster-data/master-0/wal
uuid: "0c027bf9265047459a0880a8e28508f3"
format_stamp: "Formatted at 2026-06-19 09:05:13 on dist-test-slave-mtdr"
server_key: "51b27b014da1428c3e9df0e1880528a6"
server_key_iv: "fdf7619ecd0ad5607520b3b686f3d87c"
server_key_version: "encryptionkey@0"
I20260619 09:05:13.239796 22507 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781859658134510-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781859658134510-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781859658134510-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 09:05:13.243173 22507 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 09:05:13.244915 22507 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 09:05:13.245023 22507 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 09:05:13.248855 22507 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41827
I20260619 09:05:13.248883 22574 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41827 every 8 connection(s)
I20260619 09:05:13.249176 22507 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781859658134510-392-0/minicluster-data/master-0/data/info.pb
I20260619 09:05:13.250015 22575 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 09:05:13.251421 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ez5NB/build/release/bin/kudu as pid 22507
I20260619 09:05:13.251509 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781859658134510-392-0/minicluster-data/master-0/wal/instance
I20260619 09:05:13.251685 392 external_mini_cluster.cc:1428] Setting key 7b98512b678b68a614b7dacba22f028c
I20260619 09:05:13.252327 22575 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c027bf9265047459a0880a8e28508f3: Bootstrap starting.
I20260619 09:05:13.252841 22575 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0c027bf9265047459a0880a8e28508f3: Neither blocks nor log segments found. Creating new log.
I20260619 09:05:13.253136 22575 log.cc:824] T 00000000000000000000000000000000 P 0c027bf9265047459a0880a8e28508f3: Log is configured to *not* fsync() on all Append() calls
I20260619 09:05:13.253836 22575 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c027bf9265047459a0880a8e28508f3: No bootstrap required, opened a new log
I20260619 09:05:13.253908 22578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:05:13.252116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57847 (local address 127.0.98.62:41827)
0619 09:05:13.252301 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0619 09:05:13.252308 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 09:05:13.252331 (+ 23us) server_negotiation.cc:407] Connection header received
0619 09:05:13.252747 (+ 416us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:05:13.252753 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:05:13.252779 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:05:13.252893 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:05:13.253513 (+ 620us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:05:13.253517 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:05:13.253517 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 09:05:13.253521 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:05:13.253562 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 09:05:13.253576 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:05:13.253655 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0619 09:05:13.253695 (+ 40us) server_negotiation.cc:299] Negotiation successful
0619 09:05:13.253731 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":98,"threads_started":1}
I20260619 09:05:13.254747 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 09:05:13.255897 22575 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0c027bf9265047459a0880a8e28508f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c027bf9265047459a0880a8e28508f3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41827 } }
I20260619 09:05:13.256073 22575 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0c027bf9265047459a0880a8e28508f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 09:05:13.256119 22575 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0c027bf9265047459a0880a8e28508f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c027bf9265047459a0880a8e28508f3, State: Initialized, Role: FOLLOWER
I20260619 09:05:13.256242 22575 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0c027bf9265047459a0880a8e28508f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c027bf9265047459a0880a8e28508f3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41827 } }
I20260619 09:05:13.256312 22575 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0c027bf9265047459a0880a8e28508f3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 09:05:13.256336 22575 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0c027bf9265047459a0880a8e28508f3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 09:05:13.256378 22575 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0c027bf9265047459a0880a8e28508f3 [term 0 FOLLOWER]: Advancing to term 1
I20260619 09:05:13.256851 22575 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0c027bf9265047459a0880a8e28508f3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c027bf9265047459a0880a8e28508f3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41827 } }
I20260619 09:05:13.256935 22578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:05:13.256009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47829 (local address 127.0.98.62:41827)
0619 09:05:13.256039 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0619 09:05:13.256041 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:05:13.256139 (+ 98us) server_negotiation.cc:407] Connection header received
0619 09:05:13.256207 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:05:13.256208 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:05:13.256212 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:05:13.256255 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:05:13.256743 (+ 488us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:05:13.256744 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:05:13.256744 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 09:05:13.256748 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:05:13.256759 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 09:05:13.256763 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:05:13.256814 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0619 09:05:13.256867 (+ 53us) server_negotiation.cc:299] Negotiation successful
0619 09:05:13.256879 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260619 09:05:13.256942 22575 leader_election.cc:302] T 00000000000000000000000000000000 P 0c027bf9265047459a0880a8e28508f3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0c027bf9265047459a0880a8e28508f3; no voters:
I20260619 09:05:13.257117 22575 leader_election.cc:288] T 00000000000000000000000000000000 P 0c027bf9265047459a0880a8e28508f3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 09:05:13.257154 22585 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0c027bf9265047459a0880a8e28508f3 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 09:05:13.257313 22585 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0c027bf9265047459a0880a8e28508f3 [term 1 LEADER]: Becoming Leader. State: Replica: 0c027bf9265047459a0880a8e28508f3, State: Running, Role: LEADER
I20260619 09:05:13.257354 22575 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0c027bf9265047459a0880a8e28508f3 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 09:05:13.257448 22585 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0c027bf9265047459a0880a8e28508f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c027bf9265047459a0880a8e28508f3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41827 } }
I20260619 09:05:13.257771 22587 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c027bf9265047459a0880a8e28508f3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0c027bf9265047459a0880a8e28508f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c027bf9265047459a0880a8e28508f3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41827 } } }
I20260619 09:05:13.257776 22586 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c027bf9265047459a0880a8e28508f3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0c027bf9265047459a0880a8e28508f3. Latest consensus state: current_term: 1 leader_uuid: "0c027bf9265047459a0880a8e28508f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c027bf9265047459a0880a8e28508f3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41827 } } }
I20260619 09:05:13.257912 22586 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c027bf9265047459a0880a8e28508f3 [sys.catalog]: This master's current role is: LEADER
I20260619 09:05:13.258054 22587 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c027bf9265047459a0880a8e28508f3 [sys.catalog]: This master's current role is: LEADER
I20260619 09:05:13.258127 22592 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 09:05:13.258569 22592 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 09:05:13.260025 22592 catalog_manager.cc:1347] Generated new cluster ID: 3d8825f76f7244c189d8b9f8b7cd8e7c
I20260619 09:05:13.260063 22592 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 09:05:13.269887 22592 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 09:05:13.270035 22592 catalog_manager.cc:1504] Loading token signing keys...
I20260619 09:05:13.276202 22592 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0c027bf9265047459a0880a8e28508f3: Generated new TSK 0
I20260619 09:05:13.276374 22592 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 09:05:13.284428 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ez5NB/build/release/bin/kudu with pid 22507
2026-06-19T09:05:13Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1210 ms)
[----------] 18 tests from AuthTokenIssuingTest (21886 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781859658134510-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 09:05:13 dist-test-slave-mtdr krb5kdc[22606](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 09:05:13 dist-test-slave-mtdr krb5kdc[22606](info): set up 2 sockets
Jun 19 09:05:13 dist-test-slave-mtdr krb5kdc[22606](info): commencing operation
krb5kdc: starting...
W20260619 09:05:14.230741 392 mini_kdc.cc:121] Time spent starting KDC: real 0.929s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781859658134510-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781859658134510-392-0/krb5kdc/test-user.keytab.
Jun 19 09:05:14 dist-test-slave-mtdr krb5kdc[22606](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859914, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T09:05:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T09:05:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781859658134510-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781859658134510-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 09:05:14.341743 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ez5NB/build/release/bin/kudu
/tmp/dist-test-task4ez5NB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781859658134510-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781859658134510-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781859658134510-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781859658134510-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40701
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781859658134510-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39763
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40701,127.0.98.61:46549,127.0.98.60:42379
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781859658134510-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781859658134510-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781859658134510-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 09:05:14.418776 22619 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 09:05:14.419000 22619 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 09:05:14.419015 22619 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 09:05:14.419034 22619 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 09:05:14.420089 22619 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 09:05:14.420121 22619 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 09:05:14.420133 22619 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 09:05:14.420145 22619 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 09:05:14.420154 22619 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 09:05:14.421535 22619 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39763
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781859658134510-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781859658134510-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40701,127.0.98.61:46549,127.0.98.60:42379
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781859658134510-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40701
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781859658134510-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22619
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781859658134510-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 09:05:14.421720 22619 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 09:05:14.421886 22619 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 09:05:14.426296 22627 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:05:14.426306 22625 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:05:14.426306 22624 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:05:14.426606 22619 server_base.cc:1034] running on GCE node
I20260619 09:05:14.426851 22619 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 09:05:14.427111 22619 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 09:05:14.428257 22619 hybrid_clock.cc:648] HybridClock initialized: now 1781859914428236 us; error 42 us; skew 500 ppm
Jun 19 09:05:14 dist-test-slave-mtdr krb5kdc[22606](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859914, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 09:05:14.433367 22619 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 09:05:14.433792 22619 webserver.cc:460] Webserver started at http://127.0.98.62:43195/ using document root <none> and password file <none>
I20260619 09:05:14.433986 22619 fs_manager.cc:359] Metadata directory not provided
I20260619 09:05:14.434026 22619 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 09:05:14.434113 22619 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 09:05:14.435119 22619 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781859658134510-392-0/minicluster-data/master-0/data/instance:
uuid: "45518fc3d28249e0a6309a329026cac1"
format_stamp: "Formatted at 2026-06-19 09:05:14 on dist-test-slave-mtdr"
server_key: "147347a4e7cf87992d25ddf959798015"
server_key_iv: "4345d213b390eca6d9d217c760e483f9"
server_key_version: "encryptionkey@0"
I20260619 09:05:14.435405 22619 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781859658134510-392-0/minicluster-data/master-0/wal/instance:
uuid: "45518fc3d28249e0a6309a329026cac1"
format_stamp: "Formatted at 2026-06-19 09:05:14 on dist-test-slave-mtdr"
server_key: "147347a4e7cf87992d25ddf959798015"
server_key_iv: "4345d213b390eca6d9d217c760e483f9"
server_key_version: "encryptionkey@0"
I20260619 09:05:14.436659 22619 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 09:05:14.437330 22634 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 09:05:14.437496 22619 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 09:05:14.437572 22619 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781859658134510-392-0/minicluster-data/master-0/data,/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781859658134510-392-0/minicluster-data/master-0/wal
uuid: "45518fc3d28249e0a6309a329026cac1"
format_stamp: "Formatted at 2026-06-19 09:05:14 on dist-test-slave-mtdr"
server_key: "147347a4e7cf87992d25ddf959798015"
server_key_iv: "4345d213b390eca6d9d217c760e483f9"
server_key_version: "encryptionkey@0"
I20260619 09:05:14.437636 22619 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781859658134510-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781859658134510-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781859658134510-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 09:05:14.447834 22619 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 09:05:14.449764 22619 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 09:05:14.449873 22619 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 09:05:14.453485 22619 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40701
I20260619 09:05:14.453516 22686 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40701 every 8 connection(s)
I20260619 09:05:14.453804 22619 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781859658134510-392-0/minicluster-data/master-0/data/info.pb
I20260619 09:05:14.454674 22687 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 09:05:14.456015 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ez5NB/build/release/bin/kudu as pid 22619
I20260619 09:05:14.456116 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781859658134510-392-0/minicluster-data/master-0/wal/instance
I20260619 09:05:14.456308 392 external_mini_cluster.cc:1428] Setting key 3e596d8ecde5adb3070ff7d37353aa3f
I20260619 09:05:14.456308 22687 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40701 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 09:05:14 dist-test-slave-mtdr krb5kdc[22606](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859914, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
I20260619 09:05:14.466837 22690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:05:14.457191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51782 (local address 127.0.98.62:40701)
0619 09:05:14.457292 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0619 09:05:14.457297 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 09:05:14.457326 (+ 29us) server_negotiation.cc:407] Connection header received
0619 09:05:14.457931 (+ 605us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:05:14.457934 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:05:14.457956 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:05:14.458052 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:05:14.458569 (+ 517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:05:14.459846 (+ 1277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:05:14.460665 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:05:14.461205 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:05:14.461329 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:05:14.463631 (+ 2302us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:05:14.463640 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:05:14.463640 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:05:14.463647 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:05:14.465194 (+ 1547us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:05:14.465659 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:05:14.465662 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:05:14.465663 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:05:14.465763 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:05:14.466037 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:05:14.466038 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:05:14.466039 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:05:14.466283 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:05:14.466349 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0619 09:05:14.466558 (+ 209us) server_negotiation.cc:299] Negotiation successful
0619 09:05:14.466651 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73,"thread_start_us":37,"threads_started":1}
I20260619 09:05:14.466858 22689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:05:14.457043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40701 (local address 127.0.0.1:51782)
0619 09:05:14.457198 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0619 09:05:14.457242 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 09:05:14.457776 (+ 534us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:05:14.458085 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:05:14.458088 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:05:14.458329 (+ 241us) client_negotiation.cc:189] Negotiated authn=SASL
0619 09:05:14.458478 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:05:14.458482 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:05:14.459927 (+ 1445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:05:14.459928 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:05:14.460587 (+ 659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:05:14.460589 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:05:14.461360 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:05:14.461361 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:05:14.461419 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:05:14.461910 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 09:05:14.461919 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0619 09:05:14.463532 (+ 1613us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 09:05:14.465295 (+ 1763us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:05:14.465302 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:05:14.465304 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:05:14.465575 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:05:14.465848 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:05:14.465849 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:05:14.465849 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:05:14.465972 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:05:14.466349 (+ 377us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 09:05:14.466357 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 09:05:14.466432 (+ 75us) client_negotiation.cc:769] Sending connection context
0619 09:05:14.466540 (+ 108us) client_negotiation.cc:240] Negotiation successful
0619 09:05:14.466724 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"mutex_wait_us":45,"thread_start_us":61,"threads_started":1}
I20260619 09:05:14.467535 22687 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46549 } has no permanent_uuid. Determining permanent_uuid...
I20260619 09:05:14.467855 22689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:05:14.467748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46549 (local address 127.0.0.1:43392)
0619 09:05:14.467791 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0619 09:05:14.467836 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260619 09:05:14.467988 22638 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46549: connect: Connection refused (error 111)
I20260619 09:05:14.468371 22689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:05:14.468288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46549 (local address 127.0.0.1:43394)
0619 09:05:14.468326 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0619 09:05:14.468349 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260619 09:05:14.468459 22687 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:46549: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46549: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781859658134510-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781859658134510-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260619 09:05:14.481616 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ez5NB/build/release/bin/kudu
/tmp/dist-test-task4ez5NB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781859658134510-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781859658134510-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781859658134510-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781859658134510-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:46549
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781859658134510-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39763
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40701,127.0.98.61:46549,127.0.98.60:42379
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781859658134510-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781859658134510-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781859658134510-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 09:05:14.517648 22687 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46549 } attempt: 1
I20260619 09:05:14.518093 22689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:05:14.517972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46549 (local address 127.0.0.1:43408)
0619 09:05:14.518029 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0619 09:05:14.518063 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260619 09:05:14.518471 22689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:05:14.518378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46549 (local address 127.0.0.1:43424)
0619 09:05:14.518421 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0619 09:05:14.518447 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260619 09:05:14.518575 22687 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:46549: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46549: connect: Connection refused (error 111)
W20260619 09:05:14.559244 22694 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 09:05:14.559464 22694 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 09:05:14.559489 22694 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 09:05:14.559513 22694 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 09:05:14.560616 22694 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 09:05:14.560652 22694 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 09:05:14.560670 22694 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 09:05:14.560688 22694 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 09:05:14.560702 22694 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 09:05:14.562186 22694 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39763
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781859658134510-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781859658134510-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40701,127.0.98.61:46549,127.0.98.60:42379
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781859658134510-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:46549
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781859658134510-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22694
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781859658134510-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 09:05:14.562417 22694 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 09:05:14.562614 22694 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 09:05:14.567081 22699 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:05:14.567070 22700 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:05:14.567051 22702 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:05:14.567094 22694 server_base.cc:1034] running on GCE node
I20260619 09:05:14.567500 22694 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 09:05:14.567775 22694 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 09:05:14.568936 22694 hybrid_clock.cc:648] HybridClock initialized: now 1781859914568924 us; error 36 us; skew 500 ppm
Jun 19 09:05:14 dist-test-slave-mtdr krb5kdc[22606](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859914, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 09:05:14.571377 22694 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260619 09:05:14.571791 22694 webserver.cc:460] Webserver started at http://127.0.98.61:34741/ using document root <none> and password file <none>
I20260619 09:05:14.571997 22694 fs_manager.cc:359] Metadata directory not provided
I20260619 09:05:14.572044 22694 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 09:05:14.572162 22694 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 09:05:14.573148 22694 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781859658134510-392-0/minicluster-data/master-1/data/instance:
uuid: "042c37d399ca484b9c1edd2fcdb27bc9"
format_stamp: "Formatted at 2026-06-19 09:05:14 on dist-test-slave-mtdr"
server_key: "e6456b66f87e1ba173c166edaca2faa3"
server_key_iv: "f771e33d5b6f9ac0b968bea3d546369b"
server_key_version: "encryptionkey@0"
I20260619 09:05:14.573467 22694 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781859658134510-392-0/minicluster-data/master-1/wal/instance:
uuid: "042c37d399ca484b9c1edd2fcdb27bc9"
format_stamp: "Formatted at 2026-06-19 09:05:14 on dist-test-slave-mtdr"
server_key: "e6456b66f87e1ba173c166edaca2faa3"
server_key_iv: "f771e33d5b6f9ac0b968bea3d546369b"
server_key_version: "encryptionkey@0"
I20260619 09:05:14.574676 22694 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 09:05:14.575373 22709 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 09:05:14.575548 22694 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260619 09:05:14.575630 22694 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781859658134510-392-0/minicluster-data/master-1/data,/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781859658134510-392-0/minicluster-data/master-1/wal
uuid: "042c37d399ca484b9c1edd2fcdb27bc9"
format_stamp: "Formatted at 2026-06-19 09:05:14 on dist-test-slave-mtdr"
server_key: "e6456b66f87e1ba173c166edaca2faa3"
server_key_iv: "f771e33d5b6f9ac0b968bea3d546369b"
server_key_version: "encryptionkey@0"
I20260619 09:05:14.575696 22694 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781859658134510-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781859658134510-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781859658134510-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 09:05:14.586714 22687 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46549 } attempt: 2
I20260619 09:05:14.587091 22689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:05:14.586981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46549 (local address 127.0.0.1:43440)
0619 09:05:14.587034 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0619 09:05:14.587062 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260619 09:05:14.587428 22689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:05:14.587302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46549 (local address 127.0.0.1:43452)
0619 09:05:14.587357 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0619 09:05:14.587395 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260619 09:05:14.587527 22687 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:46549: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46549: connect: Connection refused (error 111)
I20260619 09:05:14.596432 22694 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 09:05:14.598068 22694 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 09:05:14.598183 22694 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 09:05:14.601794 22694 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:46549
I20260619 09:05:14.601825 22761 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:46549 every 8 connection(s)
I20260619 09:05:14.602144 22694 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781859658134510-392-0/minicluster-data/master-1/data/info.pb
I20260619 09:05:14.603091 22762 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 09:05:14.604524 22762 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40701 } has no permanent_uuid. Determining permanent_uuid...
I20260619 09:05:14.606006 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ez5NB/build/release/bin/kudu as pid 22694
I20260619 09:05:14.606099 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781859658134510-392-0/minicluster-data/master-1/wal/instance
I20260619 09:05:14.606272 392 external_mini_cluster.cc:1428] Setting key cc6f414cd254318b59eb4cc78688d089
Jun 19 09:05:14 dist-test-slave-mtdr krb5kdc[22606](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859914, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
I20260619 09:05:14.613384 22690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:05:14.605138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51784 (local address 127.0.98.62:40701)
0619 09:05:14.605196 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0619 09:05:14.605198 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:05:14.605396 (+ 198us) server_negotiation.cc:407] Connection header received
0619 09:05:14.605867 (+ 471us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:05:14.605868 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:05:14.605880 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:05:14.605951 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:05:14.606604 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:05:14.607283 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:05:14.608234 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:05:14.608427 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:05:14.608483 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:05:14.610678 (+ 2195us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:05:14.610679 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:05:14.610680 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:05:14.610688 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:05:14.612085 (+ 1397us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:05:14.612491 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:05:14.612492 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:05:14.612493 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:05:14.612552 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:05:14.612801 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:05:14.612802 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:05:14.612802 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:05:14.612955 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:05:14.613025 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0619 09:05:14.613253 (+ 228us) server_negotiation.cc:299] Negotiation successful
0619 09:05:14.613318 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260619 09:05:14.613615 22764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:05:14.605115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40701 (local address 127.0.0.1:51784)
0619 09:05:14.605272 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0619 09:05:14.605304 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0619 09:05:14.605746 (+ 442us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:05:14.606009 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:05:14.606012 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:05:14.606298 (+ 286us) client_negotiation.cc:189] Negotiated authn=SASL
0619 09:05:14.606523 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:05:14.606528 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:05:14.607370 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:05:14.607370 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:05:14.608150 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:05:14.608153 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:05:14.608513 (+ 360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:05:14.608514 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:05:14.608626 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:05:14.609055 (+ 429us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 09:05:14.609063 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 09:05:14.610580 (+ 1517us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 09:05:14.612167 (+ 1587us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:05:14.612175 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:05:14.612177 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:05:14.612419 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:05:14.612610 (+ 191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:05:14.612611 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:05:14.612611 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:05:14.612716 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:05:14.613028 (+ 312us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 09:05:14.613033 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 09:05:14.613151 (+ 118us) client_negotiation.cc:769] Sending connection context
0619 09:05:14.613233 (+ 82us) client_negotiation.cc:240] Negotiation successful
0619 09:05:14.613366 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":83,"threads_started":1}
I20260619 09:05:14.614131 22762 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46549 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
Jun 19 09:05:14 dist-test-slave-mtdr krb5kdc[22606](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859914, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260619 09:05:14.621546 22764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:05:14.614380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46549 (local address 127.0.0.1:43468)
0619 09:05:14.614429 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0619 09:05:14.614439 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 09:05:14.614482 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:05:14.614696 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:05:14.614697 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:05:14.614877 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0619 09:05:14.614936 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:05:14.614937 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:05:14.615817 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:05:14.615818 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:05:14.616315 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:05:14.616316 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:05:14.616760 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:05:14.616760 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:05:14.616831 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:05:14.617193 (+ 362us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 09:05:14.617197 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 09:05:14.618537 (+ 1340us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 09:05:14.620526 (+ 1989us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:05:14.620527 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:05:14.620527 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:05:14.620687 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:05:14.620905 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:05:14.620906 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:05:14.620906 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:05:14.620963 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:05:14.621282 (+ 319us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 09:05:14.621283 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 09:05:14.621322 (+ 39us) client_negotiation.cc:769] Sending connection context
0619 09:05:14.621399 (+ 77us) client_negotiation.cc:240] Negotiation successful
0619 09:05:14.621483 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260619 09:05:14.621661 22766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:05:14.614420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43468 (local address 127.0.98.61:46549)
0619 09:05:14.614569 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0619 09:05:14.614574 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 09:05:14.614594 (+ 20us) server_negotiation.cc:407] Connection header received
0619 09:05:14.614612 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:05:14.614614 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:05:14.614624 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:05:14.614690 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:05:14.615011 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:05:14.615733 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:05:14.616388 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:05:14.616693 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:05:14.616756 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:05:14.618624 (+ 1868us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:05:14.618632 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:05:14.618633 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:05:14.618639 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:05:14.620368 (+ 1729us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:05:14.620766 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:05:14.620769 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:05:14.620770 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:05:14.620825 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:05:14.621016 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:05:14.621017 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:05:14.621017 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:05:14.621185 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:05:14.621273 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0619 09:05:14.621447 (+ 174us) server_negotiation.cc:299] Negotiation successful
0619 09:05:14.621545 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":19,"server-negotiator.queue_time_us":119,"thread_start_us":85,"threads_started":1}
I20260619 09:05:14.622196 22762 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42379 } has no permanent_uuid. Determining permanent_uuid...
I20260619 09:05:14.622515 22764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:05:14.622389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42379 (local address 127.0.0.1:59198)
0619 09:05:14.622447 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0619 09:05:14.622495 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260619 09:05:14.622664 22713 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42379: connect: Connection refused (error 111)
I20260619 09:05:14.623065 22764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:05:14.622972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42379 (local address 127.0.0.1:59208)
0619 09:05:14.623011 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0619 09:05:14.623047 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260619 09:05:14.623178 22762 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:42379: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42379: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781859658134510-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781859658134510-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260619 09:05:14.629927 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ez5NB/build/release/bin/kudu
/tmp/dist-test-task4ez5NB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781859658134510-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781859658134510-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781859658134510-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781859658134510-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:42379
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781859658134510-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39763
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40701,127.0.98.61:46549,127.0.98.60:42379
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781859658134510-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781859658134510-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781859658134510-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 09:05:14.672413 22762 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42379 } attempt: 1
I20260619 09:05:14.672871 22764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:05:14.672749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42379 (local address 127.0.0.1:59212)
0619 09:05:14.672802 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0619 09:05:14.672841 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260619 09:05:14.673219 22764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:05:14.673092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42379 (local address 127.0.0.1:59216)
0619 09:05:14.673148 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0619 09:05:14.673189 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260619 09:05:14.673377 22762 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:42379: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42379: connect: Connection refused (error 111)
I20260619 09:05:14.678678 22687 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46549 } attempt: 3
Jun 19 09:05:14 dist-test-slave-mtdr krb5kdc[22606](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859914, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260619 09:05:14.685326 22689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:05:14.678995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46549 (local address 127.0.0.1:43484)
0619 09:05:14.679054 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0619 09:05:14.679065 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 09:05:14.679105 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:05:14.679239 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:05:14.679239 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:05:14.679375 (+ 136us) client_negotiation.cc:189] Negotiated authn=SASL
0619 09:05:14.679443 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:05:14.679445 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:05:14.680362 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:05:14.680363 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:05:14.680881 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:05:14.680882 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:05:14.681351 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:05:14.681352 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:05:14.681412 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:05:14.681874 (+ 462us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 09:05:14.681879 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 09:05:14.683092 (+ 1213us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 09:05:14.684446 (+ 1354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:05:14.684447 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:05:14.684447 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:05:14.684606 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:05:14.684765 (+ 159us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:05:14.684765 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:05:14.684766 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:05:14.684820 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:05:14.685030 (+ 210us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 09:05:14.685031 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 09:05:14.685078 (+ 47us) client_negotiation.cc:769] Sending connection context
0619 09:05:14.685136 (+ 58us) client_negotiation.cc:240] Negotiation successful
0619 09:05:14.685231 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260619 09:05:14.685326 22766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:05:14.679034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43484 (local address 127.0.98.61:46549)
0619 09:05:14.679089 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0619 09:05:14.679091 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:05:14.679102 (+ 11us) server_negotiation.cc:407] Connection header received
0619 09:05:14.679151 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:05:14.679152 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:05:14.679164 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:05:14.679230 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:05:14.679505 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:05:14.680266 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:05:14.680952 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:05:14.681270 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:05:14.681349 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:05:14.683169 (+ 1820us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:05:14.683171 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:05:14.683171 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:05:14.683179 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:05:14.684365 (+ 1186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:05:14.684663 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:05:14.684663 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:05:14.684664 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:05:14.684712 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:05:14.684866 (+ 154us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:05:14.684867 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:05:14.684867 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:05:14.684974 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:05:14.685028 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0619 09:05:14.685162 (+ 134us) server_negotiation.cc:299] Negotiation successful
0619 09:05:14.685231 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260619 09:05:14.685729 22687 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42379 } has no permanent_uuid. Determining permanent_uuid...
I20260619 09:05:14.686010 22689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:05:14.685923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42379 (local address 127.0.0.1:59224)
0619 09:05:14.685962 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0619 09:05:14.685987 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
I20260619 09:05:14.686280 22689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:05:14.686201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42379 (local address 127.0.0.1:59234)
0619 09:05:14.686238 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0619 09:05:14.686259 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260619 09:05:14.686357 22687 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:42379: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42379: connect: Connection refused (error 111)
W20260619 09:05:14.708747 22769 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 09:05:14.708951 22769 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 09:05:14.708976 22769 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 09:05:14.709002 22769 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 09:05:14.710095 22769 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 09:05:14.710134 22769 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 09:05:14.710152 22769 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 09:05:14.710167 22769 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 09:05:14.710183 22769 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 09:05:14.711634 22769 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39763
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781859658134510-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781859658134510-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40701,127.0.98.61:46549,127.0.98.60:42379
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781859658134510-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:42379
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781859658134510-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22769
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781859658134510-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 09:05:14.711862 22769 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 09:05:14.712052 22769 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 09:05:14.716485 22775 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:05:14.716498 22777 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:05:14.716488 22774 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:05:14.716737 22769 server_base.cc:1034] running on GCE node
I20260619 09:05:14.716938 22769 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 09:05:14.717190 22769 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 09:05:14.717492 22687 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42379 } attempt: 1
I20260619 09:05:14.717859 22689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:05:14.717745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42379 (local address 127.0.0.1:59242)
0619 09:05:14.717794 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0619 09:05:14.717825 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260619 09:05:14.718138 22689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:05:14.718057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42379 (local address 127.0.0.1:59250)
0619 09:05:14.718094 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0619 09:05:14.718116 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260619 09:05:14.718212 22687 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:42379: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42379: connect: Connection refused (error 111)
I20260619 09:05:14.718318 22769 hybrid_clock.cc:648] HybridClock initialized: now 1781859914718304 us; error 29 us; skew 500 ppm
Jun 19 09:05:14 dist-test-slave-mtdr krb5kdc[22606](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859914, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 09:05:14.720629 22769 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260619 09:05:14.721050 22769 webserver.cc:460] Webserver started at http://127.0.98.60:42905/ using document root <none> and password file <none>
I20260619 09:05:14.721237 22769 fs_manager.cc:359] Metadata directory not provided
I20260619 09:05:14.721289 22769 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 09:05:14.721428 22769 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 09:05:14.722430 22769 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781859658134510-392-0/minicluster-data/master-2/data/instance:
uuid: "73166faf55b8439d9c89f5a109f3f1c1"
format_stamp: "Formatted at 2026-06-19 09:05:14 on dist-test-slave-mtdr"
server_key: "9c42b5dd3af72bb75540d8111847903e"
server_key_iv: "4fe44601f04c85894ef51ea6db1c2db7"
server_key_version: "encryptionkey@0"
I20260619 09:05:14.722745 22769 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781859658134510-392-0/minicluster-data/master-2/wal/instance:
uuid: "73166faf55b8439d9c89f5a109f3f1c1"
format_stamp: "Formatted at 2026-06-19 09:05:14 on dist-test-slave-mtdr"
server_key: "9c42b5dd3af72bb75540d8111847903e"
server_key_iv: "4fe44601f04c85894ef51ea6db1c2db7"
server_key_version: "encryptionkey@0"
I20260619 09:05:14.723929 22769 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260619 09:05:14.724666 22784 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 09:05:14.724823 22769 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260619 09:05:14.724907 22769 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781859658134510-392-0/minicluster-data/master-2/data,/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781859658134510-392-0/minicluster-data/master-2/wal
uuid: "73166faf55b8439d9c89f5a109f3f1c1"
format_stamp: "Formatted at 2026-06-19 09:05:14 on dist-test-slave-mtdr"
server_key: "9c42b5dd3af72bb75540d8111847903e"
server_key_iv: "4fe44601f04c85894ef51ea6db1c2db7"
server_key_version: "encryptionkey@0"
I20260619 09:05:14.724979 22769 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781859658134510-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781859658134510-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781859658134510-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 09:05:14.736320 22769 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 09:05:14.738139 22769 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 09:05:14.738265 22769 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 09:05:14.741528 22762 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42379 } attempt: 2
I20260619 09:05:14.741918 22764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:05:14.741779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42379 (local address 127.0.0.1:59264)
0619 09:05:14.741846 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0619 09:05:14.741885 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260619 09:05:14.742259 22764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:05:14.742167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42379 (local address 127.0.0.1:59266)
0619 09:05:14.742202 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0619 09:05:14.742230 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
I20260619 09:05:14.742280 22769 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:42379
I20260619 09:05:14.742304 22836 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:42379 every 8 connection(s)
W20260619 09:05:14.742383 22762 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:42379: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42379: connect: Connection refused (error 111)
I20260619 09:05:14.742628 22769 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781859658134510-392-0/minicluster-data/master-2/data/info.pb
I20260619 09:05:14.743515 22837 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 09:05:14.743994 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ez5NB/build/release/bin/kudu as pid 22769
I20260619 09:05:14.744084 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781859658134510-392-0/minicluster-data/master-2/wal/instance
I20260619 09:05:14.744273 392 external_mini_cluster.cc:1428] Setting key b6689ff710dd019d7f6af23b326dba14
I20260619 09:05:14.745164 22837 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40701 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 09:05:14 dist-test-slave-mtdr krb5kdc[22606](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859914, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Jun 19 09:05:14 dist-test-slave-mtdr krb5kdc[22606](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859914, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260619 09:05:14.752954 22690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:05:14.745940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51792 (local address 127.0.98.62:40701)
0619 09:05:14.745985 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0619 09:05:14.745986 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 09:05:14.746132 (+ 146us) server_negotiation.cc:407] Connection header received
0619 09:05:14.746173 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:05:14.746173 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:05:14.746187 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:05:14.746263 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:05:14.746676 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:05:14.747203 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:05:14.748021 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:05:14.748217 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:05:14.748271 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:05:14.750381 (+ 2110us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:05:14.750382 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:05:14.750383 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:05:14.750390 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:05:14.751691 (+ 1301us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:05:14.752048 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:05:14.752049 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:05:14.752049 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:05:14.752116 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:05:14.752315 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:05:14.752316 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:05:14.752316 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:05:14.752428 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:05:14.752490 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0619 09:05:14.752770 (+ 280us) server_negotiation.cc:299] Negotiation successful
0619 09:05:14.752857 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260619 09:05:14.753084 22841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:05:14.745909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40701 (local address 127.0.0.1:51792)
0619 09:05:14.746050 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0619 09:05:14.746078 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0619 09:05:14.746133 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:05:14.746279 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:05:14.746281 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:05:14.746498 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0619 09:05:14.746592 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:05:14.746596 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:05:14.747279 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:05:14.747280 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:05:14.747938 (+ 658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:05:14.747940 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:05:14.748303 (+ 363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:05:14.748305 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:05:14.748414 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:05:14.748838 (+ 424us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 09:05:14.748848 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 09:05:14.750307 (+ 1459us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 09:05:14.751764 (+ 1457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:05:14.751770 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:05:14.751772 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:05:14.751983 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:05:14.752167 (+ 184us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:05:14.752168 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:05:14.752168 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:05:14.752261 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:05:14.752481 (+ 220us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 09:05:14.752487 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 09:05:14.752637 (+ 150us) client_negotiation.cc:769] Sending connection context
0619 09:05:14.752746 (+ 109us) client_negotiation.cc:240] Negotiation successful
0619 09:05:14.752888 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"mutex_wait_us":20,"thread_start_us":97,"threads_started":1}
I20260619 09:05:14.753666 22840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:05:14.744776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59270 (local address 127.0.98.60:42379)
0619 09:05:14.744959 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0619 09:05:14.744968 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 09:05:14.744999 (+ 31us) server_negotiation.cc:407] Connection header received
0619 09:05:14.745402 (+ 403us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:05:14.745408 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:05:14.745456 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:05:14.745585 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:05:14.746361 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:05:14.747316 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:05:14.748057 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:05:14.748390 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:05:14.748454 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:05:14.750915 (+ 2461us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:05:14.750922 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:05:14.750922 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:05:14.750928 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:05:14.752501 (+ 1573us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:05:14.752857 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:05:14.752860 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:05:14.752860 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:05:14.752922 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:05:14.753132 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:05:14.753133 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:05:14.753134 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:05:14.753267 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:05:14.753366 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0619 09:05:14.753503 (+ 137us) server_negotiation.cc:299] Negotiation successful
0619 09:05:14.753577 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":130,"thread_start_us":78,"threads_started":1}
I20260619 09:05:14.753695 22837 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46549 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 09:05:14 dist-test-slave-mtdr krb5kdc[22606](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859914, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Jun 19 09:05:14 dist-test-slave-mtdr krb5kdc[22606](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859914, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 09:05:14.760272 22766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:05:14.754109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43488 (local address 127.0.98.61:46549)
0619 09:05:14.754156 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0619 09:05:14.754158 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:05:14.754168 (+ 10us) server_negotiation.cc:407] Connection header received
0619 09:05:14.754193 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:05:14.754194 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:05:14.754205 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:05:14.754267 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:05:14.754560 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:05:14.755080 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:05:14.755974 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:05:14.756283 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:05:14.756370 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:05:14.758046 (+ 1676us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:05:14.758048 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:05:14.758048 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:05:14.758055 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:05:14.759292 (+ 1237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:05:14.759639 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:05:14.759640 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:05:14.759641 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:05:14.759699 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:05:14.759875 (+ 176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:05:14.759876 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:05:14.759876 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:05:14.759973 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:05:14.760021 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0619 09:05:14.760155 (+ 134us) server_negotiation.cc:299] Negotiation successful
0619 09:05:14.760210 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260619 09:05:14.760322 22841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:05:14.754020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46549 (local address 127.0.0.1:43488)
0619 09:05:14.754063 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0619 09:05:14.754072 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 09:05:14.754105 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:05:14.754273 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:05:14.754273 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:05:14.754401 (+ 128us) client_negotiation.cc:189] Negotiated authn=SASL
0619 09:05:14.754455 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:05:14.754456 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:05:14.755154 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:05:14.755155 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:05:14.755891 (+ 736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:05:14.755893 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:05:14.756374 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:05:14.756375 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:05:14.756429 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:05:14.756843 (+ 414us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 09:05:14.756848 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 09:05:14.757970 (+ 1122us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 09:05:14.759347 (+ 1377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:05:14.759348 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:05:14.759348 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:05:14.759575 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:05:14.759756 (+ 181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:05:14.759756 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:05:14.759757 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:05:14.759813 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:05:14.760027 (+ 214us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 09:05:14.760028 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 09:05:14.760067 (+ 39us) client_negotiation.cc:769] Sending connection context
0619 09:05:14.760138 (+ 71us) client_negotiation.cc:240] Negotiation successful
0619 09:05:14.760217 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260619 09:05:14.760641 22837 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42379 } has no permanent_uuid. Determining permanent_uuid...
I20260619 09:05:14.761996 22690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:05:14.754457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51804 (local address 127.0.98.62:40701)
0619 09:05:14.754505 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0619 09:05:14.754507 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:05:14.754513 (+ 6us) server_negotiation.cc:407] Connection header received
0619 09:05:14.754530 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:05:14.754531 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:05:14.754539 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:05:14.754582 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:05:14.755059 (+ 477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:05:14.755749 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:05:14.756616 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:05:14.756883 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:05:14.756930 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:05:14.758947 (+ 2017us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:05:14.758948 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:05:14.758948 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:05:14.758954 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:05:14.760398 (+ 1444us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:05:14.760803 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:05:14.760804 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:05:14.760804 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:05:14.760861 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:05:14.761069 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:05:14.761070 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:05:14.761070 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:05:14.761168 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:05:14.761272 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0619 09:05:14.761879 (+ 607us) server_negotiation.cc:299] Negotiation successful
0619 09:05:14.761936 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
Jun 19 09:05:14 dist-test-slave-mtdr krb5kdc[22606](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859914, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Jun 19 09:05:14 dist-test-slave-mtdr krb5kdc[22606](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859914, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260619 09:05:14.766685 22841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:05:14.760842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42379 (local address 127.0.0.1:59276)
0619 09:05:14.760876 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0619 09:05:14.760884 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0619 09:05:14.760913 (+ 29us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:05:14.761016 (+ 103us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:05:14.761017 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:05:14.761143 (+ 126us) client_negotiation.cc:189] Negotiated authn=SASL
0619 09:05:14.761166 (+ 23us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:05:14.761167 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:05:14.761724 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:05:14.761724 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:05:14.762343 (+ 619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:05:14.762345 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:05:14.762857 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:05:14.762858 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:05:14.762916 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:05:14.763202 (+ 286us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 09:05:14.763206 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 09:05:14.764417 (+ 1211us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 09:05:14.765804 (+ 1387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:05:14.765805 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:05:14.765806 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:05:14.765989 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:05:14.766156 (+ 167us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:05:14.766157 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:05:14.766157 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:05:14.766200 (+ 43us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:05:14.766397 (+ 197us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 09:05:14.766397 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 09:05:14.766427 (+ 30us) client_negotiation.cc:769] Sending connection context
0619 09:05:14.766480 (+ 53us) client_negotiation.cc:240] Negotiation successful
0619 09:05:14.766589 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23}
I20260619 09:05:14.766747 22840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:05:14.760880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59276 (local address 127.0.98.60:42379)
0619 09:05:14.760909 (+ 29us) server_negotiation.cc:206] Beginning negotiation
0619 09:05:14.760911 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:05:14.760918 (+ 7us) server_negotiation.cc:407] Connection header received
0619 09:05:14.760968 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:05:14.760969 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:05:14.760977 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:05:14.761013 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:05:14.761226 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:05:14.761664 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:05:14.762408 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:05:14.762652 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:05:14.762712 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:05:14.764494 (+ 1782us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:05:14.764495 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:05:14.764495 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:05:14.764501 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:05:14.765730 (+ 1229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:05:14.766049 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:05:14.766050 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:05:14.766050 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:05:14.766098 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:05:14.766236 (+ 138us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:05:14.766236 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:05:14.766237 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:05:14.766346 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:05:14.766403 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0619 09:05:14.766514 (+ 111us) server_negotiation.cc:299] Negotiation successful
0619 09:05:14.766649 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":72,"server-negotiator.queue_time_us":16}
I20260619 09:05:14.768146 22837 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 73166faf55b8439d9c89f5a109f3f1c1: Bootstrap starting.
I20260619 09:05:14.768909 22837 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 73166faf55b8439d9c89f5a109f3f1c1: Neither blocks nor log segments found. Creating new log.
I20260619 09:05:14.769028 22766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:05:14.762681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43494 (local address 127.0.98.61:46549)
0619 09:05:14.762721 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0619 09:05:14.762723 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:05:14.762729 (+ 6us) server_negotiation.cc:407] Connection header received
0619 09:05:14.762745 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:05:14.762746 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:05:14.762753 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:05:14.762796 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:05:14.763328 (+ 532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:05:14.763868 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:05:14.764641 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:05:14.764846 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:05:14.764895 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:05:14.766923 (+ 2028us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:05:14.766924 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:05:14.766925 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:05:14.766931 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:05:14.768083 (+ 1152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:05:14.768399 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:05:14.768400 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:05:14.768400 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:05:14.768452 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:05:14.768603 (+ 151us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:05:14.768604 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:05:14.768604 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:05:14.768703 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:05:14.768849 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0619 09:05:14.768881 (+ 32us) server_negotiation.cc:299] Negotiation successful
0619 09:05:14.768945 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260619 09:05:14.769194 22837 log.cc:824] T 00000000000000000000000000000000 P 73166faf55b8439d9c89f5a109f3f1c1: Log is configured to *not* fsync() on all Append() calls
I20260619 09:05:14.769938 22837 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 73166faf55b8439d9c89f5a109f3f1c1: No bootstrap required, opened a new log
I20260619 09:05:14.770354 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 09:05:14.771570 22837 raft_consensus.cc:348] T 00000000000000000000000000000000 P 73166faf55b8439d9c89f5a109f3f1c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45518fc3d28249e0a6309a329026cac1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40701 } } peers { permanent_uuid: "042c37d399ca484b9c1edd2fcdb27bc9" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46549 } } peers { permanent_uuid: "73166faf55b8439d9c89f5a109f3f1c1" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42379 } }
I20260619 09:05:14.771692 22837 raft_consensus.cc:729] T 00000000000000000000000000000000 P 73166faf55b8439d9c89f5a109f3f1c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73166faf55b8439d9c89f5a109f3f1c1, State: Initialized, Role: FOLLOWER
I20260619 09:05:14.771767 22837 consensus_queue.cc:260] T 00000000000000000000000000000000 P 73166faf55b8439d9c89f5a109f3f1c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45518fc3d28249e0a6309a329026cac1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40701 } } peers { permanent_uuid: "042c37d399ca484b9c1edd2fcdb27bc9" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46549 } } peers { permanent_uuid: "73166faf55b8439d9c89f5a109f3f1c1" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42379 } }
I20260619 09:05:14.772038 22843 sys_catalog.cc:455] T 00000000000000000000000000000000 P 73166faf55b8439d9c89f5a109f3f1c1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45518fc3d28249e0a6309a329026cac1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40701 } } peers { permanent_uuid: "042c37d399ca484b9c1edd2fcdb27bc9" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46549 } } peers { permanent_uuid: "73166faf55b8439d9c89f5a109f3f1c1" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42379 } } }
I20260619 09:05:14.772092 22837 sys_catalog.cc:564] T 00000000000000000000000000000000 P 73166faf55b8439d9c89f5a109f3f1c1 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 09:05:14.772130 22843 sys_catalog.cc:458] T 00000000000000000000000000000000 P 73166faf55b8439d9c89f5a109f3f1c1 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 09:05:14.771417 22661 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "45518fc3d28249e0a6309a329026cac1"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51804
W20260619 09:05:14.773639 22854 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 73166faf55b8439d9c89f5a109f3f1c1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 09:05:14.773725 22854 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 09:05:14.773924 22769 master_runner.cc:186] Error getting master registration for 127.0.98.62:40701: OK, instance_id { permanent_uuid: "45518fc3d28249e0a6309a329026cac1" instance_seqno: 1781859914452500 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 09:05:14.776643 22769 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 09:05:14.793346 22687 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42379 } attempt: 2
Jun 19 09:05:14 dist-test-slave-mtdr krb5kdc[22606](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859914, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260619 09:05:14.800491 22840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:05:14.793738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59288 (local address 127.0.98.60:42379)
0619 09:05:14.793799 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0619 09:05:14.793802 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 09:05:14.793810 (+ 8us) server_negotiation.cc:407] Connection header received
0619 09:05:14.793833 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:05:14.793834 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:05:14.793842 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:05:14.793889 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:05:14.794180 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:05:14.794966 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:05:14.795620 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:05:14.795920 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:05:14.795983 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:05:14.797883 (+ 1900us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:05:14.797884 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:05:14.797884 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:05:14.797892 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:05:14.799369 (+ 1477us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:05:14.799738 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:05:14.799739 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:05:14.799739 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:05:14.799790 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:05:14.799953 (+ 163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:05:14.799954 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:05:14.799954 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:05:14.800128 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:05:14.800187 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0619 09:05:14.800323 (+ 136us) server_negotiation.cc:299] Negotiation successful
0619 09:05:14.800389 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260619 09:05:14.800489 22689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:05:14.793661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42379 (local address 127.0.0.1:59288)
0619 09:05:14.793715 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0619 09:05:14.793727 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 09:05:14.793764 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:05:14.793899 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:05:14.793900 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:05:14.794062 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0619 09:05:14.794109 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:05:14.794110 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:05:14.795037 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:05:14.795038 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:05:14.795530 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:05:14.795532 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:05:14.795992 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:05:14.795993 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:05:14.796042 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:05:14.796502 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 09:05:14.796507 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 09:05:14.797800 (+ 1293us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 09:05:14.799448 (+ 1648us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:05:14.799449 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:05:14.799449 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:05:14.799671 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:05:14.799843 (+ 172us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:05:14.799843 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:05:14.799844 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:05:14.799903 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:05:14.800185 (+ 282us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 09:05:14.800186 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 09:05:14.800235 (+ 49us) client_negotiation.cc:769] Sending connection context
0619 09:05:14.800298 (+ 63us) client_negotiation.cc:240] Negotiation successful
0619 09:05:14.800389 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19}
I20260619 09:05:14.802088 22687 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 45518fc3d28249e0a6309a329026cac1: Bootstrap starting.
I20260619 09:05:14.802850 22687 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 45518fc3d28249e0a6309a329026cac1: Neither blocks nor log segments found. Creating new log.
I20260619 09:05:14.803185 22687 log.cc:824] T 00000000000000000000000000000000 P 45518fc3d28249e0a6309a329026cac1: Log is configured to *not* fsync() on all Append() calls
I20260619 09:05:14.803977 22687 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 45518fc3d28249e0a6309a329026cac1: No bootstrap required, opened a new log
I20260619 09:05:14.805331 22687 raft_consensus.cc:348] T 00000000000000000000000000000000 P 45518fc3d28249e0a6309a329026cac1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45518fc3d28249e0a6309a329026cac1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40701 } } peers { permanent_uuid: "042c37d399ca484b9c1edd2fcdb27bc9" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46549 } } peers { permanent_uuid: "73166faf55b8439d9c89f5a109f3f1c1" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42379 } }
I20260619 09:05:14.805467 22687 raft_consensus.cc:729] T 00000000000000000000000000000000 P 45518fc3d28249e0a6309a329026cac1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45518fc3d28249e0a6309a329026cac1, State: Initialized, Role: FOLLOWER
I20260619 09:05:14.805545 22687 consensus_queue.cc:260] T 00000000000000000000000000000000 P 45518fc3d28249e0a6309a329026cac1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45518fc3d28249e0a6309a329026cac1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40701 } } peers { permanent_uuid: "042c37d399ca484b9c1edd2fcdb27bc9" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46549 } } peers { permanent_uuid: "73166faf55b8439d9c89f5a109f3f1c1" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42379 } }
I20260619 09:05:14.805850 22687 sys_catalog.cc:564] T 00000000000000000000000000000000 P 45518fc3d28249e0a6309a329026cac1 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 09:05:14.805893 22661 raft_consensus.cc:482] T 00000000000000000000000000000000 P 45518fc3d28249e0a6309a329026cac1 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260619 09:05:14.805932 22661 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 45518fc3d28249e0a6309a329026cac1 [term 0 FOLLOWER]: Advancing to term 1
I20260619 09:05:14.806653 22661 raft_consensus.cc:504] T 00000000000000000000000000000000 P 45518fc3d28249e0a6309a329026cac1 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45518fc3d28249e0a6309a329026cac1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40701 } } peers { permanent_uuid: "042c37d399ca484b9c1edd2fcdb27bc9" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46549 } } peers { permanent_uuid: "73166faf55b8439d9c89f5a109f3f1c1" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42379 } }
I20260619 09:05:14.806865 22661 leader_election.cc:288] T 00000000000000000000000000000000 P 45518fc3d28249e0a6309a329026cac1 [CANDIDATE]: Term 1 election: Requested vote from peers 042c37d399ca484b9c1edd2fcdb27bc9 (127.0.98.61:46549), 73166faf55b8439d9c89f5a109f3f1c1 (127.0.98.60:42379)
I20260619 09:05:14.807258 22811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "45518fc3d28249e0a6309a329026cac1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "73166faf55b8439d9c89f5a109f3f1c1"
I20260619 09:05:14.807386 22811 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 73166faf55b8439d9c89f5a109f3f1c1 [term 0 FOLLOWER]: Advancing to term 1
W20260619 09:05:14.807809 22868 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 45518fc3d28249e0a6309a329026cac1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 09:05:14.807860 22868 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 09:05:14.808239 22811 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 73166faf55b8439d9c89f5a109f3f1c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 45518fc3d28249e0a6309a329026cac1 in term 1.
I20260619 09:05:14.805826 22857 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45518fc3d28249e0a6309a329026cac1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45518fc3d28249e0a6309a329026cac1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40701 } } peers { permanent_uuid: "042c37d399ca484b9c1edd2fcdb27bc9" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46549 } } peers { permanent_uuid: "73166faf55b8439d9c89f5a109f3f1c1" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42379 } } }
I20260619 09:05:14.808509 22857 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45518fc3d28249e0a6309a329026cac1 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 09:05:14.809827 22638 leader_election.cc:302] T 00000000000000000000000000000000 P 45518fc3d28249e0a6309a329026cac1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45518fc3d28249e0a6309a329026cac1, 73166faf55b8439d9c89f5a109f3f1c1; no voters:
I20260619 09:05:14.810103 22619 master_runner.cc:186] Error getting master registration for 127.0.98.61:46549: OK, instance_id { permanent_uuid: "042c37d399ca484b9c1edd2fcdb27bc9" instance_seqno: 1781859914600850 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 09:05:14.812347 22619 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 09:05:14.812410 22857 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 45518fc3d28249e0a6309a329026cac1 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 09:05:14.812544 22857 raft_consensus.cc:686] T 00000000000000000000000000000000 P 45518fc3d28249e0a6309a329026cac1 [term 1 LEADER]: Becoming Leader. State: Replica: 45518fc3d28249e0a6309a329026cac1, State: Running, Role: LEADER
I20260619 09:05:14.812640 22857 consensus_queue.cc:237] T 00000000000000000000000000000000 P 45518fc3d28249e0a6309a329026cac1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45518fc3d28249e0a6309a329026cac1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40701 } } peers { permanent_uuid: "042c37d399ca484b9c1edd2fcdb27bc9" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46549 } } peers { permanent_uuid: "73166faf55b8439d9c89f5a109f3f1c1" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42379 } }
I20260619 09:05:14.813055 22857 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45518fc3d28249e0a6309a329026cac1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 45518fc3d28249e0a6309a329026cac1. Latest consensus state: current_term: 1 leader_uuid: "45518fc3d28249e0a6309a329026cac1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45518fc3d28249e0a6309a329026cac1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40701 } } peers { permanent_uuid: "042c37d399ca484b9c1edd2fcdb27bc9" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46549 } } peers { permanent_uuid: "73166faf55b8439d9c89f5a109f3f1c1" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42379 } } }
I20260619 09:05:14.813144 22857 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45518fc3d28249e0a6309a329026cac1 [sys.catalog]: This master's current role is: LEADER
I20260619 09:05:14.814529 22876 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 09:05:14.814695 22876 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 09:05:14.824476 22766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:05:14.813450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43500 (local address 127.0.98.61:46549)
0619 09:05:14.813493 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0619 09:05:14.813495 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:05:14.815056 (+ 1561us) server_negotiation.cc:407] Connection header received
0619 09:05:14.815191 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:05:14.815192 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:05:14.815202 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:05:14.815232 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:05:14.816375 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:05:14.817059 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:05:14.818427 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:05:14.818754 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:05:14.818792 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:05:14.820924 (+ 2132us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:05:14.820925 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:05:14.820925 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:05:14.820931 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:05:14.822369 (+ 1438us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:05:14.822922 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:05:14.822923 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:05:14.822923 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:05:14.822980 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:05:14.823415 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:05:14.823416 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:05:14.823417 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:05:14.823550 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:05:14.824023 (+ 473us) server_negotiation.cc:1035] Waiting for connection context
0619 09:05:14.824311 (+ 288us) server_negotiation.cc:299] Negotiation successful
0619 09:05:14.824385 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260619 09:05:14.826449 22811 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 73166faf55b8439d9c89f5a109f3f1c1 [term 1 FOLLOWER]: Refusing update from remote peer 45518fc3d28249e0a6309a329026cac1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 09:05:14.826802 22874 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 45518fc3d28249e0a6309a329026cac1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "73166faf55b8439d9c89f5a109f3f1c1" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42379 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 09:05:14.829010 22843 sys_catalog.cc:455] T 00000000000000000000000000000000 P 73166faf55b8439d9c89f5a109f3f1c1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 45518fc3d28249e0a6309a329026cac1. Latest consensus state: current_term: 1 leader_uuid: "45518fc3d28249e0a6309a329026cac1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45518fc3d28249e0a6309a329026cac1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40701 } } peers { permanent_uuid: "042c37d399ca484b9c1edd2fcdb27bc9" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46549 } } peers { permanent_uuid: "73166faf55b8439d9c89f5a109f3f1c1" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42379 } } }
I20260619 09:05:14.829113 22843 sys_catalog.cc:458] T 00000000000000000000000000000000 P 73166faf55b8439d9c89f5a109f3f1c1 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 09:05:14.830881 22840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:05:14.813800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59300 (local address 127.0.98.60:42379)
0619 09:05:14.813847 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0619 09:05:14.813849 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:05:14.815525 (+ 1676us) server_negotiation.cc:407] Connection header received
0619 09:05:14.815612 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:05:14.815613 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:05:14.815622 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:05:14.815650 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:05:14.816312 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:05:14.817050 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:05:14.818581 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:05:14.818874 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:05:14.818908 (+ 34us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:05:14.820787 (+ 1879us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:05:14.820789 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:05:14.820789 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:05:14.820795 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:05:14.822980 (+ 2185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:05:14.829687 (+ 6707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:05:14.829688 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:05:14.829688 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:05:14.829755 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:05:14.830151 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:05:14.830152 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:05:14.830152 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:05:14.830294 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:05:14.830361 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0619 09:05:14.830671 (+ 310us) server_negotiation.cc:299] Negotiation successful
0619 09:05:14.830752 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260619 09:05:14.831401 22690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:05:14.814136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51816 (local address 127.0.98.62:40701)
0619 09:05:14.814170 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0619 09:05:14.814173 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 09:05:14.822961 (+ 8788us) server_negotiation.cc:407] Connection header received
0619 09:05:14.823060 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:05:14.823061 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:05:14.823071 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:05:14.823099 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:05:14.823705 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:05:14.824379 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:05:14.825264 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:05:14.825524 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:05:14.825562 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:05:14.827603 (+ 2041us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:05:14.827604 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:05:14.827604 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:05:14.827612 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:05:14.829453 (+ 1841us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:05:14.830107 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:05:14.830108 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:05:14.830109 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:05:14.830175 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:05:14.830445 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:05:14.830446 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:05:14.830446 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:05:14.830589 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:05:14.830665 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0619 09:05:14.831238 (+ 573us) server_negotiation.cc:299] Negotiation successful
0619 09:05:14.831323 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260619 09:05:14.833092 22857 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45518fc3d28249e0a6309a329026cac1 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "45518fc3d28249e0a6309a329026cac1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45518fc3d28249e0a6309a329026cac1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40701 } } peers { permanent_uuid: "042c37d399ca484b9c1edd2fcdb27bc9" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46549 } } peers { permanent_uuid: "73166faf55b8439d9c89f5a109f3f1c1" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42379 } } }
I20260619 09:05:14.833173 22857 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45518fc3d28249e0a6309a329026cac1 [sys.catalog]: This master's current role is: LEADER
I20260619 09:05:14.833242 22843 sys_catalog.cc:455] T 00000000000000000000000000000000 P 73166faf55b8439d9c89f5a109f3f1c1 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "45518fc3d28249e0a6309a329026cac1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45518fc3d28249e0a6309a329026cac1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40701 } } peers { permanent_uuid: "042c37d399ca484b9c1edd2fcdb27bc9" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46549 } } peers { permanent_uuid: "73166faf55b8439d9c89f5a109f3f1c1" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42379 } } }
I20260619 09:05:14.833316 22843 sys_catalog.cc:458] T 00000000000000000000000000000000 P 73166faf55b8439d9c89f5a109f3f1c1 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 09:05:14.833827 22880 mvcc.cc:205] Tried to move back new op lower bound from 7298498211092336640 to 7298498211073155072. Current Snapshot: MvccSnapshot[applied={T|T < 7298498211092336640}]
I20260619 09:05:14.834342 22876 catalog_manager.cc:1347] Generated new cluster ID: a4ac4a37176d424ea44ae588f12920bf
I20260619 09:05:14.834388 22876 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 09:05:14.836095 22762 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42379 } attempt: 3
I20260619 09:05:14.840070 22881 mvcc.cc:205] Tried to move back new op lower bound from 7298498211092336640 to 7298498211073155072. Current Snapshot: MvccSnapshot[applied={T|T < 7298498211092336640}]
Jun 19 09:05:14 dist-test-slave-mtdr krb5kdc[22606](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859914, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260619 09:05:14.845903 22764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:05:14.836331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42379 (local address 127.0.0.1:59302)
0619 09:05:14.836371 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0619 09:05:14.836380 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 09:05:14.836410 (+ 30us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:05:14.839030 (+ 2620us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:05:14.839031 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:05:14.839197 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0619 09:05:14.839250 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:05:14.839253 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:05:14.840151 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:05:14.840152 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:05:14.840860 (+ 708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:05:14.840862 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:05:14.841443 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:05:14.841444 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:05:14.841495 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:05:14.841945 (+ 450us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 09:05:14.841962 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 09:05:14.843352 (+ 1390us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 09:05:14.844943 (+ 1591us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:05:14.844944 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:05:14.844944 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:05:14.845146 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:05:14.845312 (+ 166us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:05:14.845313 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:05:14.845313 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:05:14.845368 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:05:14.845611 (+ 243us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 09:05:14.845612 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 09:05:14.845652 (+ 40us) client_negotiation.cc:769] Sending connection context
0619 09:05:14.845714 (+ 62us) client_negotiation.cc:240] Negotiation successful
0619 09:05:14.845803 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25}
I20260619 09:05:14.845903 22840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:05:14.838864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59302 (local address 127.0.98.60:42379)
0619 09:05:14.838900 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0619 09:05:14.838902 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:05:14.838909 (+ 7us) server_negotiation.cc:407] Connection header received
0619 09:05:14.838927 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:05:14.838927 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:05:14.838936 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:05:14.838987 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:05:14.839345 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:05:14.840030 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:05:14.841087 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:05:14.841381 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:05:14.841435 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:05:14.843436 (+ 2001us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:05:14.843437 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:05:14.843437 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:05:14.843444 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:05:14.844870 (+ 1426us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:05:14.845196 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:05:14.845197 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:05:14.845197 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:05:14.845247 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:05:14.845430 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:05:14.845430 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:05:14.845431 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:05:14.845565 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:05:14.845620 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0619 09:05:14.845745 (+ 125us) server_negotiation.cc:299] Negotiation successful
0619 09:05:14.845820 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260619 09:05:14.851056 22762 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 042c37d399ca484b9c1edd2fcdb27bc9: Bootstrap starting.
I20260619 09:05:14.851809 22762 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 042c37d399ca484b9c1edd2fcdb27bc9: Neither blocks nor log segments found. Creating new log.
I20260619 09:05:14.852118 22762 log.cc:824] T 00000000000000000000000000000000 P 042c37d399ca484b9c1edd2fcdb27bc9: Log is configured to *not* fsync() on all Append() calls
I20260619 09:05:14.854485 22876 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 09:05:14.855218 22876 catalog_manager.cc:1504] Loading token signing keys...
I20260619 09:05:14.860484 22762 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 042c37d399ca484b9c1edd2fcdb27bc9: No bootstrap required, opened a new log
I20260619 09:05:14.862589 22762 raft_consensus.cc:348] T 00000000000000000000000000000000 P 042c37d399ca484b9c1edd2fcdb27bc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45518fc3d28249e0a6309a329026cac1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40701 } } peers { permanent_uuid: "042c37d399ca484b9c1edd2fcdb27bc9" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46549 } } peers { permanent_uuid: "73166faf55b8439d9c89f5a109f3f1c1" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42379 } }
I20260619 09:05:14.862723 22762 raft_consensus.cc:729] T 00000000000000000000000000000000 P 042c37d399ca484b9c1edd2fcdb27bc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 042c37d399ca484b9c1edd2fcdb27bc9, State: Initialized, Role: FOLLOWER
I20260619 09:05:14.862803 22762 consensus_queue.cc:260] T 00000000000000000000000000000000 P 042c37d399ca484b9c1edd2fcdb27bc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45518fc3d28249e0a6309a329026cac1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40701 } } peers { permanent_uuid: "042c37d399ca484b9c1edd2fcdb27bc9" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46549 } } peers { permanent_uuid: "73166faf55b8439d9c89f5a109f3f1c1" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42379 } }
I20260619 09:05:14.863260 22736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "45518fc3d28249e0a6309a329026cac1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "042c37d399ca484b9c1edd2fcdb27bc9"
I20260619 09:05:14.863317 22735 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 042c37d399ca484b9c1edd2fcdb27bc9 [term 0 FOLLOWER]: Advancing to term 1
I20260619 09:05:14.864378 22735 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 042c37d399ca484b9c1edd2fcdb27bc9 [term 1 FOLLOWER]: Refusing update from remote peer 45518fc3d28249e0a6309a329026cac1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 09:05:14.864943 22857 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 45518fc3d28249e0a6309a329026cac1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "042c37d399ca484b9c1edd2fcdb27bc9" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46549 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 09:05:14.865284 22736 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 042c37d399ca484b9c1edd2fcdb27bc9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 45518fc3d28249e0a6309a329026cac1 in term 1.
I20260619 09:05:14.863188 22762 sys_catalog.cc:564] T 00000000000000000000000000000000 P 042c37d399ca484b9c1edd2fcdb27bc9 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 09:05:14.867365 22876 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 45518fc3d28249e0a6309a329026cac1: Generated new TSK 0
I20260619 09:05:14.867532 22876 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 09:05:14.867718 22895 sys_catalog.cc:455] T 00000000000000000000000000000000 P 042c37d399ca484b9c1edd2fcdb27bc9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 45518fc3d28249e0a6309a329026cac1. Latest consensus state: current_term: 1 leader_uuid: "45518fc3d28249e0a6309a329026cac1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45518fc3d28249e0a6309a329026cac1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40701 } } peers { permanent_uuid: "042c37d399ca484b9c1edd2fcdb27bc9" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46549 } } peers { permanent_uuid: "73166faf55b8439d9c89f5a109f3f1c1" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42379 } } }
I20260619 09:05:14.867898 22895 sys_catalog.cc:458] T 00000000000000000000000000000000 P 042c37d399ca484b9c1edd2fcdb27bc9 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 09:05:14.868713 22874 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45518fc3d28249e0a6309a329026cac1 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "45518fc3d28249e0a6309a329026cac1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45518fc3d28249e0a6309a329026cac1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40701 } } peers { permanent_uuid: "042c37d399ca484b9c1edd2fcdb27bc9" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46549 } } peers { permanent_uuid: "73166faf55b8439d9c89f5a109f3f1c1" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42379 } } }
I20260619 09:05:14.868803 22874 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45518fc3d28249e0a6309a329026cac1 [sys.catalog]: This master's current role is: LEADER
I20260619 09:05:14.863184 22887 sys_catalog.cc:455] T 00000000000000000000000000000000 P 042c37d399ca484b9c1edd2fcdb27bc9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45518fc3d28249e0a6309a329026cac1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40701 } } peers { permanent_uuid: "042c37d399ca484b9c1edd2fcdb27bc9" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46549 } } peers { permanent_uuid: "73166faf55b8439d9c89f5a109f3f1c1" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42379 } } }
I20260619 09:05:14.869060 22887 sys_catalog.cc:458] T 00000000000000000000000000000000 P 042c37d399ca484b9c1edd2fcdb27bc9 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 09:05:14.869603 22889 mvcc.cc:205] Tried to move back new op lower bound from 7298498211283902464 to 7298498211073155072. Current Snapshot: MvccSnapshot[applied={T|T < 7298498211092336640}]
I20260619 09:05:14.868049 22895 sys_catalog.cc:455] T 00000000000000000000000000000000 P 042c37d399ca484b9c1edd2fcdb27bc9 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "45518fc3d28249e0a6309a329026cac1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45518fc3d28249e0a6309a329026cac1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40701 } } peers { permanent_uuid: "042c37d399ca484b9c1edd2fcdb27bc9" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46549 } } peers { permanent_uuid: "73166faf55b8439d9c89f5a109f3f1c1" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42379 } } }
I20260619 09:05:14.869854 22895 sys_catalog.cc:458] T 00000000000000000000000000000000 P 042c37d399ca484b9c1edd2fcdb27bc9 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 09:05:14.871436 22902 catalog_manager.cc:1259] Loaded cluster ID: a4ac4a37176d424ea44ae588f12920bf
I20260619 09:05:14.871538 22902 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 042c37d399ca484b9c1edd2fcdb27bc9: loading cluster ID for follower catalog manager: success
I20260619 09:05:14.872686 22902 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 042c37d399ca484b9c1edd2fcdb27bc9: acquiring CA information for follower catalog manager: success
I20260619 09:05:14.873035 22902 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 042c37d399ca484b9c1edd2fcdb27bc9: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260619 09:05:14.894496 22766 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 09:05:14.892968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43516 (local address 127.0.98.61:46549)
0619 09:05:14.893013 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0619 09:05:14.893015 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:05:14.893871 (+ 856us) server_negotiation.cc:407] Connection header received
0619 09:05:14.893968 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:05:14.893969 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:05:14.893981 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:05:14.894010 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:05:14.894427 (+ 417us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43516: BlockingRecv error: recv got EOF from 127.0.0.1:43516 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260619 09:05:14.895537 22690 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 09:05:14.893624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51826 (local address 127.0.98.62:40701)
0619 09:05:14.893660 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0619 09:05:14.893662 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:05:14.895027 (+ 1365us) server_negotiation.cc:407] Connection header received
0619 09:05:14.895133 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:05:14.895134 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:05:14.895143 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:05:14.895172 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:05:14.895477 (+ 305us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51826: BlockingRecv error: recv got EOF from 127.0.0.1:51826 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260619 09:05:14.896275 22840 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 09:05:14.893309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59318 (local address 127.0.98.60:42379)
0619 09:05:14.893347 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0619 09:05:14.893349 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:05:14.894962 (+ 1613us) server_negotiation.cc:407] Connection header received
0619 09:05:14.895840 (+ 878us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:05:14.895841 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:05:14.895850 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:05:14.895877 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:05:14.896213 (+ 336us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59318: BlockingRecv error: recv got EOF from 127.0.0.1:59318 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
I20260619 09:05:14.901415 22766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:05:14.897905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43530 (local address 127.0.98.61:46549)
0619 09:05:14.897949 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0619 09:05:14.897950 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 09:05:14.898170 (+ 220us) server_negotiation.cc:407] Connection header received
0619 09:05:14.898267 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:05:14.898268 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:05:14.898305 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:05:14.898345 (+ 40us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 09:05:14.898535 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:05:14.899333 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:05:14.900458 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:05:14.900751 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:05:14.900792 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:05:14.901045 (+ 253us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 09:05:14.901177 (+ 132us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 09:05:14.901224 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0619 09:05:14.901329 (+ 105us) server_negotiation.cc:299] Negotiation successful
0619 09:05:14.901348 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
W20260619 09:05:14.904703 22840 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 09:05:14.903911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59328 (local address 127.0.98.60:42379)
0619 09:05:14.903945 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0619 09:05:14.903947 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:05:14.904195 (+ 248us) server_negotiation.cc:407] Connection header received
0619 09:05:14.904297 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:05:14.904299 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:05:14.904317 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:05:14.904345 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:05:14.904656 (+ 311us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59328: BlockingRecv error: recv got EOF from 127.0.0.1:59328 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
I20260619 09:05:14.910125 22766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:05:14.906912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43534 (local address 127.0.98.61:46549)
0619 09:05:14.906942 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0619 09:05:14.906943 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 09:05:14.907178 (+ 235us) server_negotiation.cc:407] Connection header received
0619 09:05:14.907261 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:05:14.907262 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:05:14.907270 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:05:14.907295 (+ 25us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 09:05:14.907442 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:05:14.908165 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:05:14.909279 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:05:14.909570 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:05:14.909609 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:05:14.909794 (+ 185us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 09:05:14.909827 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 09:05:14.909851 (+ 24us) server_negotiation.cc:1035] Waiting for connection context
0619 09:05:14.910018 (+ 167us) server_negotiation.cc:299] Negotiation successful
0619 09:05:14.910038 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
W20260619 09:05:14.913132 22840 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 09:05:14.912405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59340 (local address 127.0.98.60:42379)
0619 09:05:14.912438 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0619 09:05:14.912440 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:05:14.912644 (+ 204us) server_negotiation.cc:407] Connection header received
0619 09:05:14.912728 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:05:14.912729 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:05:14.912739 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:05:14.912765 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:05:14.913082 (+ 317us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59340: BlockingRecv error: recv got EOF from 127.0.0.1:59340 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
I20260619 09:05:14.919569 22766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:05:14.916238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43550 (local address 127.0.98.61:46549)
0619 09:05:14.916278 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0619 09:05:14.916279 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 09:05:14.916505 (+ 226us) server_negotiation.cc:407] Connection header received
0619 09:05:14.916595 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:05:14.916596 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:05:14.916603 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:05:14.916629 (+ 26us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 09:05:14.916776 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:05:14.917497 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:05:14.918774 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:05:14.919110 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:05:14.919148 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:05:14.919337 (+ 189us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 09:05:14.919367 (+ 30us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 09:05:14.919391 (+ 24us) server_negotiation.cc:1035] Waiting for connection context
0619 09:05:14.919484 (+ 93us) server_negotiation.cc:299] Negotiation successful
0619 09:05:14.919500 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
W20260619 09:05:14.922535 22840 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 09:05:14.921727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59348 (local address 127.0.98.60:42379)
0619 09:05:14.921761 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0619 09:05:14.921763 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:05:14.921961 (+ 198us) server_negotiation.cc:407] Connection header received
0619 09:05:14.922046 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:05:14.922047 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:05:14.922057 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:05:14.922114 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:05:14.922413 (+ 299us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59348: BlockingRecv error: recv got EOF from 127.0.0.1:59348 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
I20260619 09:05:14.930904 22766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:05:14.927689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43552 (local address 127.0.98.61:46549)
0619 09:05:14.927722 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0619 09:05:14.927724 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:05:14.928012 (+ 288us) server_negotiation.cc:407] Connection header received
0619 09:05:14.928135 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:05:14.928136 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:05:14.928145 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:05:14.928171 (+ 26us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 09:05:14.928322 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:05:14.929063 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:05:14.930154 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:05:14.930395 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:05:14.930428 (+ 33us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:05:14.930608 (+ 180us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 09:05:14.930638 (+ 30us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 09:05:14.930663 (+ 25us) server_negotiation.cc:1035] Waiting for connection context
0619 09:05:14.930778 (+ 115us) server_negotiation.cc:299] Negotiation successful
0619 09:05:14.930796 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
W20260619 09:05:14.933926 22840 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 09:05:14.933211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59350 (local address 127.0.98.60:42379)
0619 09:05:14.933253 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0619 09:05:14.933255 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:05:14.933456 (+ 201us) server_negotiation.cc:407] Connection header received
0619 09:05:14.933542 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:05:14.933544 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:05:14.933553 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:05:14.933580 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:05:14.933875 (+ 295us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59350: BlockingRecv error: recv got EOF from 127.0.0.1:59350 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
I20260619 09:05:14.946102 22766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:05:14.943084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43566 (local address 127.0.98.61:46549)
0619 09:05:14.943123 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0619 09:05:14.943125 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:05:14.943367 (+ 242us) server_negotiation.cc:407] Connection header received
0619 09:05:14.943456 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:05:14.943457 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:05:14.943465 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:05:14.943492 (+ 27us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 09:05:14.943650 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:05:14.944348 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:05:14.945371 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:05:14.945627 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:05:14.945667 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:05:14.945849 (+ 182us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 09:05:14.945880 (+ 31us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 09:05:14.945905 (+ 25us) server_negotiation.cc:1035] Waiting for connection context
0619 09:05:14.946002 (+ 97us) server_negotiation.cc:299] Negotiation successful
0619 09:05:14.946020 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
W20260619 09:05:14.949246 22840 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 09:05:14.948424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59362 (local address 127.0.98.60:42379)
0619 09:05:14.948465 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0619 09:05:14.948468 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 09:05:14.948706 (+ 238us) server_negotiation.cc:407] Connection header received
0619 09:05:14.948803 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:05:14.948804 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:05:14.948815 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:05:14.948844 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:05:14.949196 (+ 352us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59362: BlockingRecv error: recv got EOF from 127.0.0.1:59362 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
I20260619 09:05:14.970052 22766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:05:14.966728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43574 (local address 127.0.98.61:46549)
0619 09:05:14.966775 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0619 09:05:14.966777 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:05:14.966975 (+ 198us) server_negotiation.cc:407] Connection header received
0619 09:05:14.967052 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:05:14.967053 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:05:14.967065 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:05:14.967115 (+ 50us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 09:05:14.967317 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:05:14.968069 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:05:14.969254 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:05:14.969555 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:05:14.969629 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:05:14.969805 (+ 176us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 09:05:14.969839 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 09:05:14.969891 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0619 09:05:14.969955 (+ 64us) server_negotiation.cc:299] Negotiation successful
0619 09:05:14.969974 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
W20260619 09:05:14.975539 22840 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 09:05:14.972115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59372 (local address 127.0.98.60:42379)
0619 09:05:14.972157 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0619 09:05:14.972159 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:05:14.974996 (+ 2837us) server_negotiation.cc:407] Connection header received
0619 09:05:14.975051 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:05:14.975052 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:05:14.975066 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:05:14.975129 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:05:14.975468 (+ 339us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59372: BlockingRecv error: recv got EOF from 127.0.0.1:59372 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
I20260619 09:05:15.011384 22766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:05:15.008746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43584 (local address 127.0.98.61:46549)
0619 09:05:15.008811 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0619 09:05:15.008814 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 09:05:15.008871 (+ 57us) server_negotiation.cc:407] Connection header received
0619 09:05:15.008918 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:05:15.008919 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:05:15.008929 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:05:15.008998 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 09:05:15.009110 (+ 112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:05:15.009869 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:05:15.010596 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:05:15.010920 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:05:15.010981 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:05:15.011106 (+ 125us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 09:05:15.011138 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 09:05:15.011181 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0619 09:05:15.011280 (+ 99us) server_negotiation.cc:299] Negotiation successful
0619 09:05:15.011297 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
W20260619 09:05:15.013871 22840 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 09:05:15.012956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59384 (local address 127.0.98.60:42379)
0619 09:05:15.013016 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0619 09:05:15.013018 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:05:15.013289 (+ 271us) server_negotiation.cc:407] Connection header received
0619 09:05:15.013319 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:05:15.013320 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:05:15.013332 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:05:15.013426 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:05:15.013806 (+ 380us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59384: BlockingRecv error: recv got EOF from 127.0.0.1:59384 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
I20260619 09:05:15.081010 22766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:05:15.078924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43594 (local address 127.0.98.61:46549)
0619 09:05:15.078975 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0619 09:05:15.078977 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:05:15.078989 (+ 12us) server_negotiation.cc:407] Connection header received
0619 09:05:15.079063 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:05:15.079063 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:05:15.079098 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:05:15.079164 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 09:05:15.079284 (+ 120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:05:15.079766 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:05:15.080481 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:05:15.080674 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:05:15.080736 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:05:15.080831 (+ 95us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 09:05:15.080857 (+ 26us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 09:05:15.080891 (+ 34us) server_negotiation.cc:1035] Waiting for connection context
0619 09:05:15.080931 (+ 40us) server_negotiation.cc:299] Negotiation successful
0619 09:05:15.080948 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
W20260619 09:05:15.083138 22840 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 09:05:15.082519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59400 (local address 127.0.98.60:42379)
0619 09:05:15.082592 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0619 09:05:15.082595 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 09:05:15.082633 (+ 38us) server_negotiation.cc:407] Connection header received
0619 09:05:15.082705 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:05:15.082706 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:05:15.082717 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:05:15.082761 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:05:15.083069 (+ 308us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59400: BlockingRecv error: recv got EOF from 127.0.0.1:59400 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 09:05:15.214315 22766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:05:15.212137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43606 (local address 127.0.98.61:46549)
0619 09:05:15.212201 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0619 09:05:15.212203 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:05:15.212269 (+ 66us) server_negotiation.cc:407] Connection header received
0619 09:05:15.212332 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:05:15.212333 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:05:15.212345 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:05:15.212390 (+ 45us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 09:05:15.212509 (+ 119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:05:15.212979 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:05:15.213718 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:05:15.213898 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:05:15.213952 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:05:15.214082 (+ 130us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 09:05:15.214121 (+ 39us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 09:05:15.214160 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0619 09:05:15.214215 (+ 55us) server_negotiation.cc:299] Negotiation successful
0619 09:05:15.214235 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
W20260619 09:05:15.216244 22840 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 09:05:15.215739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59404 (local address 127.0.98.60:42379)
0619 09:05:15.215812 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0619 09:05:15.215814 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:05:15.215823 (+ 9us) server_negotiation.cc:407] Connection header received
0619 09:05:15.215850 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:05:15.215851 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:05:15.215861 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:05:15.215906 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:05:15.216192 (+ 286us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59404: BlockingRecv error: recv got EOF from 127.0.0.1:59404 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
I20260619 09:05:15.475498 22766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:05:15.473247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43612 (local address 127.0.98.61:46549)
0619 09:05:15.473307 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0619 09:05:15.473309 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:05:15.473369 (+ 60us) server_negotiation.cc:407] Connection header received
0619 09:05:15.473465 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:05:15.473467 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:05:15.473477 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:05:15.473529 (+ 52us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 09:05:15.473718 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:05:15.474204 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:05:15.474939 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:05:15.475129 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:05:15.475190 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:05:15.475314 (+ 124us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 09:05:15.475344 (+ 30us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 09:05:15.475387 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0619 09:05:15.475422 (+ 35us) server_negotiation.cc:299] Negotiation successful
0619 09:05:15.475439 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
W20260619 09:05:15.477567 22840 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 09:05:15.477034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59418 (local address 127.0.98.60:42379)
0619 09:05:15.477076 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0619 09:05:15.477078 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:05:15.477090 (+ 12us) server_negotiation.cc:407] Connection header received
0619 09:05:15.477143 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:05:15.477143 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:05:15.477152 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:05:15.477194 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:05:15.477509 (+ 315us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59418: BlockingRecv error: recv got EOF from 127.0.0.1:59418 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
I20260619 09:05:15.774189 22854 catalog_manager.cc:1259] Loaded cluster ID: a4ac4a37176d424ea44ae588f12920bf
I20260619 09:05:15.774272 22854 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 73166faf55b8439d9c89f5a109f3f1c1: loading cluster ID for follower catalog manager: success
I20260619 09:05:15.775146 22854 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 73166faf55b8439d9c89f5a109f3f1c1: acquiring CA information for follower catalog manager: success
I20260619 09:05:15.775481 22854 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 73166faf55b8439d9c89f5a109f3f1c1: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 09:05:15.992847 23021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:05:15.990629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43626 (local address 127.0.98.61:46549)
0619 09:05:15.990738 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0619 09:05:15.990740 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:05:15.990750 (+ 10us) server_negotiation.cc:407] Connection header received
0619 09:05:15.990843 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:05:15.990844 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:05:15.990856 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:05:15.990911 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 09:05:15.991024 (+ 113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:05:15.991512 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:05:15.992246 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:05:15.992437 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:05:15.992491 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:05:15.992608 (+ 117us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 09:05:15.992639 (+ 31us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 09:05:15.992687 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0619 09:05:15.992738 (+ 51us) server_negotiation.cc:299] Negotiation successful
0619 09:05:15.992759 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90,"thread_start_us":51,"threads_started":1}
I20260619 09:05:15.996547 23027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:05:15.994347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59422 (local address 127.0.98.60:42379)
0619 09:05:15.994439 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0619 09:05:15.994440 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 09:05:15.994448 (+ 8us) server_negotiation.cc:407] Connection header received
0619 09:05:15.994511 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:05:15.994512 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:05:15.994552 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:05:15.994614 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 09:05:15.994711 (+ 97us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:05:15.995206 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:05:15.995898 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:05:15.996075 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:05:15.996115 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:05:15.996214 (+ 99us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 09:05:15.996314 (+ 100us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 09:05:15.996365 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0619 09:05:15.996451 (+ 86us) server_negotiation.cc:299] Negotiation successful
0619 09:05:15.996468 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77,"thread_start_us":49,"threads_started":1}
I20260619 09:05:16.000845 23021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:05:15.998025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43634 (local address 127.0.98.61:46549)
0619 09:05:15.998065 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0619 09:05:15.998067 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:05:15.998154 (+ 87us) server_negotiation.cc:407] Connection header received
0619 09:05:15.998253 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:05:15.998254 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:05:15.998264 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:05:15.998288 (+ 24us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 09:05:15.998567 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:05:15.999306 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:05:16.000107 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:05:16.000394 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:05:16.000458 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:05:16.000602 (+ 144us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 09:05:16.000635 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 09:05:16.000661 (+ 26us) server_negotiation.cc:1035] Waiting for connection context
0619 09:05:16.000748 (+ 87us) server_negotiation.cc:299] Negotiation successful
0619 09:05:16.000773 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260619 09:05:16.000905 23027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:05:15.998187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59432 (local address 127.0.98.60:42379)
0619 09:05:15.998226 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0619 09:05:15.998228 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:05:15.998368 (+ 140us) server_negotiation.cc:407] Connection header received
0619 09:05:15.998452 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:05:15.998453 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:05:15.998463 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:05:15.998525 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 09:05:15.998617 (+ 92us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:05:15.999126 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:05:16.000141 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:05:16.000414 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:05:16.000463 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:05:16.000652 (+ 189us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 09:05:16.000686 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 09:05:16.000731 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0619 09:05:16.000816 (+ 85us) server_negotiation.cc:299] Negotiation successful
0619 09:05:16.000835 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260619 09:05:16.001302 23034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:05:15.998064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51836 (local address 127.0.98.62:40701)
0619 09:05:15.998166 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0619 09:05:15.998168 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:05:15.998177 (+ 9us) server_negotiation.cc:407] Connection header received
0619 09:05:15.998219 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:05:15.998220 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:05:15.998271 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:05:15.998353 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 09:05:15.998471 (+ 118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:05:15.999296 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:05:16.000275 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:05:16.000501 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:05:16.000563 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:05:16.000726 (+ 163us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 09:05:16.000829 (+ 103us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 09:05:16.001186 (+ 357us) server_negotiation.cc:1035] Waiting for connection context
0619 09:05:16.001204 (+ 18us) server_negotiation.cc:299] Negotiation successful
0619 09:05:16.001226 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82,"thread_start_us":50,"threads_started":1}
I20260619 09:05:16.002657 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ez5NB/build/release/bin/kudu with pid 22619
I20260619 09:05:16.007436 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ez5NB/build/release/bin/kudu with pid 22694
I20260619 09:05:16.011687 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ez5NB/build/release/bin/kudu with pid 22769
2026-06-19T09:05:16Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (2728 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781859658134510-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 09:05:16 dist-test-slave-mtdr krb5kdc[23038](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 09:05:16 dist-test-slave-mtdr krb5kdc[23038](info): set up 2 sockets
Jun 19 09:05:16 dist-test-slave-mtdr krb5kdc[23038](info): commencing operation
krb5kdc: starting...
W20260619 09:05:16.962769 392 mini_kdc.cc:121] Time spent starting KDC: real 0.933s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781859658134510-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781859658134510-392-0/krb5kdc/test-user.keytab.
Jun 19 09:05:17 dist-test-slave-mtdr krb5kdc[23038](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859917, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T09:05:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T09:05:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781859658134510-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781859658134510-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 09:05:17.067237 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ez5NB/build/release/bin/kudu
/tmp/dist-test-task4ez5NB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781859658134510-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781859658134510-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781859658134510-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781859658134510-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36297
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781859658134510-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36297,127.0.98.61:41743,127.0.98.60:40739
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781859658134510-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781859658134510-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781859658134510-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 09:05:17.144572 23051 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 09:05:17.144770 23051 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 09:05:17.144786 23051 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 09:05:17.144804 23051 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 09:05:17.145840 23051 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 09:05:17.145871 23051 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 09:05:17.145884 23051 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 09:05:17.145895 23051 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 09:05:17.145911 23051 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 09:05:17.147327 23051 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781859658134510-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781859658134510-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36297,127.0.98.61:41743,127.0.98.60:40739
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781859658134510-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36297
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781859658134510-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23051
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781859658134510-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 09:05:17.147514 23051 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 09:05:17.147676 23051 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 09:05:17.152132 23051 server_base.cc:1034] running on GCE node
W20260619 09:05:17.152096 23057 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:05:17.152128 23059 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:05:17.152103 23056 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:05:17.152508 23051 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 09:05:17.152777 23051 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 09:05:17.153920 23051 hybrid_clock.cc:648] HybridClock initialized: now 1781859917153903 us; error 27 us; skew 500 ppm
Jun 19 09:05:17 dist-test-slave-mtdr krb5kdc[23038](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859917, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 09:05:17.156133 23051 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 09:05:17.156507 23051 webserver.cc:460] Webserver started at http://127.0.98.62:44215/ using document root <none> and password file <none>
I20260619 09:05:17.156687 23051 fs_manager.cc:359] Metadata directory not provided
I20260619 09:05:17.156728 23051 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 09:05:17.156847 23051 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 09:05:17.157788 23051 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781859658134510-392-0/minicluster-data/master-0/data/instance:
uuid: "b38d0e4b903340a595f25851b5eefdf5"
format_stamp: "Formatted at 2026-06-19 09:05:17 on dist-test-slave-mtdr"
server_key: "3cf965b9596c44a7c918d4a5752db321"
server_key_iv: "42d8090c18f7aba2af8bd9fc3b935cb3"
server_key_version: "encryptionkey@0"
I20260619 09:05:17.158106 23051 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781859658134510-392-0/minicluster-data/master-0/wal/instance:
uuid: "b38d0e4b903340a595f25851b5eefdf5"
format_stamp: "Formatted at 2026-06-19 09:05:17 on dist-test-slave-mtdr"
server_key: "3cf965b9596c44a7c918d4a5752db321"
server_key_iv: "42d8090c18f7aba2af8bd9fc3b935cb3"
server_key_version: "encryptionkey@0"
I20260619 09:05:17.159313 23051 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 09:05:17.159952 23066 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 09:05:17.160104 23051 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 09:05:17.160184 23051 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781859658134510-392-0/minicluster-data/master-0/data,/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781859658134510-392-0/minicluster-data/master-0/wal
uuid: "b38d0e4b903340a595f25851b5eefdf5"
format_stamp: "Formatted at 2026-06-19 09:05:17 on dist-test-slave-mtdr"
server_key: "3cf965b9596c44a7c918d4a5752db321"
server_key_iv: "42d8090c18f7aba2af8bd9fc3b935cb3"
server_key_version: "encryptionkey@0"
I20260619 09:05:17.160235 23051 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781859658134510-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781859658134510-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781859658134510-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 09:05:17.171762 23051 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 09:05:17.173286 23051 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 09:05:17.173393 23051 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 09:05:17.176931 23051 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36297
I20260619 09:05:17.176975 23118 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36297 every 8 connection(s)
I20260619 09:05:17.177254 23051 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781859658134510-392-0/minicluster-data/master-0/data/info.pb
I20260619 09:05:17.178156 23119 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 09:05:17.179632 23119 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36297 } has no permanent_uuid. Determining permanent_uuid...
I20260619 09:05:17.181425 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ez5NB/build/release/bin/kudu as pid 23051
I20260619 09:05:17.181536 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781859658134510-392-0/minicluster-data/master-0/wal/instance
I20260619 09:05:17.181727 392 external_mini_cluster.cc:1428] Setting key 16d34f9373466e8de332fe8f5f07990b
Jun 19 09:05:17 dist-test-slave-mtdr krb5kdc[23038](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859917, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
I20260619 09:05:17.188766 23121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:05:17.180195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36297 (local address 127.0.0.1:48168)
0619 09:05:17.180334 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0619 09:05:17.180365 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0619 09:05:17.180814 (+ 449us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:05:17.181057 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:05:17.181060 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:05:17.181337 (+ 277us) client_negotiation.cc:189] Negotiated authn=SASL
0619 09:05:17.181550 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:05:17.181555 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:05:17.182579 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:05:17.182580 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:05:17.183390 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:05:17.183392 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:05:17.183990 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:05:17.183991 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:05:17.184038 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:05:17.184462 (+ 424us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 09:05:17.184469 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0619 09:05:17.185907 (+ 1438us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 09:05:17.187423 (+ 1516us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:05:17.187431 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:05:17.187433 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:05:17.187679 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:05:17.187883 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:05:17.187884 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:05:17.187885 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:05:17.187990 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:05:17.188280 (+ 290us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 09:05:17.188286 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 09:05:17.188371 (+ 85us) client_negotiation.cc:769] Sending connection context
0619 09:05:17.188473 (+ 102us) client_negotiation.cc:240] Negotiation successful
0619 09:05:17.188651 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"mutex_wait_us":42,"thread_start_us":61,"threads_started":1}
I20260619 09:05:17.188771 23122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:05:17.180347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48168 (local address 127.0.98.62:36297)
0619 09:05:17.180453 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0619 09:05:17.180458 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 09:05:17.180488 (+ 30us) server_negotiation.cc:407] Connection header received
0619 09:05:17.180948 (+ 460us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:05:17.180950 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:05:17.180976 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:05:17.181047 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:05:17.181642 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:05:17.182487 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:05:17.183461 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:05:17.183904 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:05:17.184009 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:05:17.185987 (+ 1978us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:05:17.185993 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:05:17.185994 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:05:17.185999 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:05:17.187346 (+ 1347us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:05:17.187751 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:05:17.187753 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:05:17.187753 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:05:17.187823 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:05:17.188046 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:05:17.188046 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:05:17.188047 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:05:17.188220 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:05:17.188277 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0619 09:05:17.188498 (+ 221us) server_negotiation.cc:299] Negotiation successful
0619 09:05:17.188589 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81,"thread_start_us":39,"threads_started":1}
I20260619 09:05:17.189407 23119 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41743 } has no permanent_uuid. Determining permanent_uuid...
I20260619 09:05:17.189693 23121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:05:17.189604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41743 (local address 127.0.0.1:51388)
0619 09:05:17.189639 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0619 09:05:17.189676 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:41743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
W20260619 09:05:17.189821 23070 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:41743: connect: Connection refused (error 111)
I20260619 09:05:17.190176 23121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:05:17.190107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41743 (local address 127.0.0.1:51394)
0619 09:05:17.190137 (+ 30us) negotiation.cc:107] Waiting for socket to connect
0619 09:05:17.190160 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:41743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
W20260619 09:05:17.190294 23119 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:41743: Network error: Client connection negotiation failed: client connection to 127.0.98.61:41743: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781859658134510-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781859658134510-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260619 09:05:17.203514 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ez5NB/build/release/bin/kudu
/tmp/dist-test-task4ez5NB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781859658134510-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781859658134510-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781859658134510-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781859658134510-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:41743
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781859658134510-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36297,127.0.98.61:41743,127.0.98.60:40739
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781859658134510-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781859658134510-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781859658134510-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 09:05:17.239533 23119 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41743 } attempt: 1
I20260619 09:05:17.240062 23121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:05:17.239900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41743 (local address 127.0.0.1:51402)
0619 09:05:17.239983 (+ 83us) negotiation.cc:107] Waiting for socket to connect
0619 09:05:17.240030 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:41743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260619 09:05:17.240432 23121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:05:17.240343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41743 (local address 127.0.0.1:51418)
0619 09:05:17.240385 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0619 09:05:17.240410 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:41743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260619 09:05:17.240523 23119 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:41743: Network error: Client connection negotiation failed: client connection to 127.0.98.61:41743: connect: Connection refused (error 111)
W20260619 09:05:17.280458 23126 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 09:05:17.280661 23126 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 09:05:17.280678 23126 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 09:05:17.280696 23126 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 09:05:17.281766 23126 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 09:05:17.281800 23126 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 09:05:17.281812 23126 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 09:05:17.281823 23126 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 09:05:17.281833 23126 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 09:05:17.283300 23126 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781859658134510-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781859658134510-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36297,127.0.98.61:41743,127.0.98.60:40739
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781859658134510-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:41743
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781859658134510-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23126
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781859658134510-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 09:05:17.283495 23126 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 09:05:17.283655 23126 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 09:05:17.288142 23134 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:05:17.288142 23131 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:05:17.288275 23126 server_base.cc:1034] running on GCE node
W20260619 09:05:17.288358 23132 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:05:17.288574 23126 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 09:05:17.288825 23126 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 09:05:17.289970 23126 hybrid_clock.cc:648] HybridClock initialized: now 1781859917289959 us; error 28 us; skew 500 ppm
Jun 19 09:05:17 dist-test-slave-mtdr krb5kdc[23038](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859917, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 09:05:17.292342 23126 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260619 09:05:17.292728 23126 webserver.cc:460] Webserver started at http://127.0.98.61:43581/ using document root <none> and password file <none>
I20260619 09:05:17.292930 23126 fs_manager.cc:359] Metadata directory not provided
I20260619 09:05:17.292977 23126 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 09:05:17.293093 23126 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 09:05:17.294068 23126 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781859658134510-392-0/minicluster-data/master-1/data/instance:
uuid: "9ef5d197ff2c472690a74e8b1809b29a"
format_stamp: "Formatted at 2026-06-19 09:05:17 on dist-test-slave-mtdr"
server_key: "347d194d689065b5f06880ca88c06be5"
server_key_iv: "35748b1237fa86ddf9dd1aa61cf89d98"
server_key_version: "encryptionkey@0"
I20260619 09:05:17.294404 23126 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781859658134510-392-0/minicluster-data/master-1/wal/instance:
uuid: "9ef5d197ff2c472690a74e8b1809b29a"
format_stamp: "Formatted at 2026-06-19 09:05:17 on dist-test-slave-mtdr"
server_key: "347d194d689065b5f06880ca88c06be5"
server_key_iv: "35748b1237fa86ddf9dd1aa61cf89d98"
server_key_version: "encryptionkey@0"
I20260619 09:05:17.295713 23126 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260619 09:05:17.296366 23141 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 09:05:17.296516 23126 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260619 09:05:17.296607 23126 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781859658134510-392-0/minicluster-data/master-1/data,/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781859658134510-392-0/minicluster-data/master-1/wal
uuid: "9ef5d197ff2c472690a74e8b1809b29a"
format_stamp: "Formatted at 2026-06-19 09:05:17 on dist-test-slave-mtdr"
server_key: "347d194d689065b5f06880ca88c06be5"
server_key_iv: "35748b1237fa86ddf9dd1aa61cf89d98"
server_key_version: "encryptionkey@0"
I20260619 09:05:17.296665 23126 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781859658134510-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781859658134510-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781859658134510-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 09:05:17.306033 23126 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 09:05:17.307790 23126 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 09:05:17.307911 23126 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 09:05:17.308673 23119 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41743 } attempt: 2
I20260619 09:05:17.308993 23121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:05:17.308897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41743 (local address 127.0.0.1:51430)
0619 09:05:17.308943 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0619 09:05:17.308968 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:41743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260619 09:05:17.309294 23121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:05:17.309217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41743 (local address 127.0.0.1:51440)
0619 09:05:17.309254 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0619 09:05:17.309278 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:41743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260619 09:05:17.309373 23119 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:41743: Network error: Client connection negotiation failed: client connection to 127.0.98.61:41743: connect: Connection refused (error 111)
I20260619 09:05:17.311637 23126 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:41743
I20260619 09:05:17.311679 23193 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:41743 every 8 connection(s)
I20260619 09:05:17.311975 23126 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781859658134510-392-0/minicluster-data/master-1/data/info.pb
I20260619 09:05:17.312877 23194 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 09:05:17.314299 23194 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36297 } has no permanent_uuid. Determining permanent_uuid...
I20260619 09:05:17.317744 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ez5NB/build/release/bin/kudu as pid 23126
I20260619 09:05:17.317830 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781859658134510-392-0/minicluster-data/master-1/wal/instance
I20260619 09:05:17.317981 392 external_mini_cluster.cc:1428] Setting key 1e57336742ba4f9fda42aae0a2ea41cf
Jun 19 09:05:17 dist-test-slave-mtdr krb5kdc[23038](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859917, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 09:05:17.323304 23122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:05:17.314907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48178 (local address 127.0.98.62:36297)
0619 09:05:17.314949 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0619 09:05:17.314951 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:05:17.315173 (+ 222us) server_negotiation.cc:407] Connection header received
0619 09:05:17.315581 (+ 408us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:05:17.315582 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:05:17.315593 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:05:17.315641 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:05:17.316221 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:05:17.316690 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:05:17.317703 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:05:17.317957 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:05:17.318020 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:05:17.320416 (+ 2396us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:05:17.320417 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:05:17.320418 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:05:17.320425 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:05:17.321921 (+ 1496us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:05:17.322347 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:05:17.322348 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:05:17.322348 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:05:17.322414 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:05:17.322690 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:05:17.322691 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:05:17.322692 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:05:17.322846 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:05:17.322916 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0619 09:05:17.323131 (+ 215us) server_negotiation.cc:299] Negotiation successful
0619 09:05:17.323217 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260619 09:05:17.323376 23196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:05:17.314924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36297 (local address 127.0.0.1:48178)
0619 09:05:17.315053 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0619 09:05:17.315085 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0619 09:05:17.315511 (+ 426us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:05:17.315695 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:05:17.315698 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:05:17.315918 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0619 09:05:17.316142 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:05:17.316146 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:05:17.316758 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:05:17.316759 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:05:17.317586 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:05:17.317589 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:05:17.318054 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:05:17.318056 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:05:17.318222 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:05:17.318652 (+ 430us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 09:05:17.318662 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 09:05:17.320324 (+ 1662us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 09:05:17.322027 (+ 1703us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:05:17.322033 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:05:17.322034 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:05:17.322262 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:05:17.322503 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:05:17.322503 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:05:17.322504 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:05:17.322637 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:05:17.322905 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 09:05:17.322909 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 09:05:17.323016 (+ 107us) client_negotiation.cc:769] Sending connection context
0619 09:05:17.323096 (+ 80us) client_negotiation.cc:240] Negotiation successful
0619 09:05:17.323204 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":56,"threads_started":1}
I20260619 09:05:17.323868 23194 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41743 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
Jun 19 09:05:17 dist-test-slave-mtdr krb5kdc[23038](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859917, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260619 09:05:17.331070 23196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:05:17.324079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41743 (local address 127.0.0.1:51446)
0619 09:05:17.324125 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0619 09:05:17.324135 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 09:05:17.324185 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:05:17.324379 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:05:17.324380 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:05:17.324508 (+ 128us) client_negotiation.cc:189] Negotiated authn=SASL
0619 09:05:17.324567 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:05:17.324569 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:05:17.325403 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:05:17.325405 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:05:17.325976 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:05:17.325979 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:05:17.326435 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:05:17.326436 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:05:17.326489 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:05:17.326791 (+ 302us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 09:05:17.326795 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 09:05:17.328452 (+ 1657us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 09:05:17.330095 (+ 1643us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:05:17.330096 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:05:17.330096 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:05:17.330245 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:05:17.330406 (+ 161us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:05:17.330407 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:05:17.330407 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:05:17.330458 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:05:17.330731 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 09:05:17.330732 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 09:05:17.330769 (+ 37us) client_negotiation.cc:769] Sending connection context
0619 09:05:17.330869 (+ 100us) client_negotiation.cc:240] Negotiation successful
0619 09:05:17.330962 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260619 09:05:17.331130 23198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:05:17.324136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51446 (local address 127.0.98.61:41743)
0619 09:05:17.324258 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0619 09:05:17.324262 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 09:05:17.324279 (+ 17us) server_negotiation.cc:407] Connection header received
0619 09:05:17.324298 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:05:17.324299 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:05:17.324309 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:05:17.324364 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:05:17.324664 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:05:17.325325 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:05:17.326058 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:05:17.326370 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:05:17.326434 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:05:17.328535 (+ 2101us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:05:17.328543 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:05:17.328543 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:05:17.328550 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:05:17.330024 (+ 1474us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:05:17.330306 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:05:17.330309 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:05:17.330309 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:05:17.330355 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:05:17.330505 (+ 150us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:05:17.330505 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:05:17.330506 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:05:17.330675 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:05:17.330746 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0619 09:05:17.330899 (+ 153us) server_negotiation.cc:299] Negotiation successful
0619 09:05:17.331012 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":50,"server-negotiator.queue_time_us":96,"thread_start_us":61,"threads_started":1}
I20260619 09:05:17.331691 23194 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40739 } has no permanent_uuid. Determining permanent_uuid...
I20260619 09:05:17.332020 23196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:05:17.331886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40739 (local address 127.0.0.1:58816)
0619 09:05:17.331934 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0619 09:05:17.331991 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260619 09:05:17.332160 23142 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40739: connect: Connection refused (error 111)
I20260619 09:05:17.332545 23196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:05:17.332446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40739 (local address 127.0.0.1:58828)
0619 09:05:17.332495 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0619 09:05:17.332524 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260619 09:05:17.332655 23194 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:40739: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40739: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781859658134510-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781859658134510-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260619 09:05:17.341470 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task4ez5NB/build/release/bin/kudu
/tmp/dist-test-task4ez5NB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781859658134510-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781859658134510-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781859658134510-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781859658134510-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:40739
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781859658134510-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36297,127.0.98.61:41743,127.0.98.60:40739
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781859658134510-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781859658134510-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781859658134510-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 09:05:17.381886 23194 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40739 } attempt: 1
I20260619 09:05:17.382388 23196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:05:17.382236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40739 (local address 127.0.0.1:58842)
0619 09:05:17.382297 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0619 09:05:17.382353 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260619 09:05:17.382710 23196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:05:17.382614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40739 (local address 127.0.0.1:58844)
0619 09:05:17.382657 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0619 09:05:17.382687 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260619 09:05:17.382836 23194 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:40739: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40739: connect: Connection refused (error 111)
I20260619 09:05:17.400504 23119 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41743 } attempt: 3
Jun 19 09:05:17 dist-test-slave-mtdr krb5kdc[23038](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859917, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260619 09:05:17.407829 23121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:05:17.400871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41743 (local address 127.0.0.1:51454)
0619 09:05:17.400959 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0619 09:05:17.400987 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0619 09:05:17.401050 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:05:17.401199 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:05:17.401199 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:05:17.401358 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0619 09:05:17.401427 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:05:17.401429 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:05:17.402313 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:05:17.402314 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:05:17.402902 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:05:17.402904 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:05:17.403370 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:05:17.403371 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:05:17.403433 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:05:17.403926 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 09:05:17.403931 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 09:05:17.405137 (+ 1206us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 09:05:17.406924 (+ 1787us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:05:17.406925 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:05:17.406925 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:05:17.407106 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:05:17.407308 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:05:17.407309 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:05:17.407309 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:05:17.407359 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:05:17.407603 (+ 244us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 09:05:17.407604 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 09:05:17.407642 (+ 38us) client_negotiation.cc:769] Sending connection context
0619 09:05:17.407692 (+ 50us) client_negotiation.cc:240] Negotiation successful
0619 09:05:17.407761 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61}
I20260619 09:05:17.407888 23198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:05:17.400955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51454 (local address 127.0.98.61:41743)
0619 09:05:17.401020 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0619 09:05:17.401022 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:05:17.401037 (+ 15us) server_negotiation.cc:407] Connection header received
0619 09:05:17.401110 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:05:17.401110 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:05:17.401125 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:05:17.401187 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:05:17.401489 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:05:17.402201 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:05:17.402971 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:05:17.403292 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:05:17.403355 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:05:17.405228 (+ 1873us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:05:17.405229 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:05:17.405230 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:05:17.405237 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:05:17.406851 (+ 1614us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:05:17.407176 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:05:17.407177 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:05:17.407177 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:05:17.407243 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:05:17.407410 (+ 167us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:05:17.407411 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:05:17.407412 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:05:17.407543 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:05:17.407610 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0619 09:05:17.407718 (+ 108us) server_negotiation.cc:299] Negotiation successful
0619 09:05:17.407800 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260619 09:05:17.408421 23119 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40739 } has no permanent_uuid. Determining permanent_uuid...
I20260619 09:05:17.408748 23121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:05:17.408655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40739 (local address 127.0.0.1:58860)
0619 09:05:17.408698 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0619 09:05:17.408725 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260619 09:05:17.409042 23121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:05:17.408983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40739 (local address 127.0.0.1:58870)
0619 09:05:17.409007 (+ 24us) negotiation.cc:107] Waiting for socket to connect
0619 09:05:17.409028 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":15}
W20260619 09:05:17.409122 23119 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:40739: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40739: connect: Connection refused (error 111)
W20260619 09:05:17.419981 23201 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 09:05:17.420161 23201 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 09:05:17.420176 23201 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 09:05:17.420197 23201 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 09:05:17.421288 23201 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 09:05:17.421325 23201 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 09:05:17.421337 23201 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 09:05:17.421348 23201 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 09:05:17.421360 23201 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 09:05:17.422847 23201 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781859658134510-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781859658134510-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36297,127.0.98.61:41743,127.0.98.60:40739
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781859658134510-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:40739
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781859658134510-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23201
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781859658134510-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 08:01:05 UTC on ccycloud.quasar-zjmztc.root.comops.site
build id 4455298
I20260619 09:05:17.423086 23201 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 09:05:17.423281 23201 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 09:05:17.427918 23207 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:05:17.427889 23206 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 09:05:17.427871 23209 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 09:05:17.428084 23201 server_base.cc:1034] running on GCE node
I20260619 09:05:17.428287 23201 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 09:05:17.428512 23201 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 09:05:17.429672 23201 hybrid_clock.cc:648] HybridClock initialized: now 1781859917429647 us; error 36 us; skew 500 ppm
Jun 19 09:05:17 dist-test-slave-mtdr krb5kdc[23038](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859917, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 09:05:17.432022 23201 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260619 09:05:17.432396 23201 webserver.cc:460] Webserver started at http://127.0.98.60:35713/ using document root <none> and password file <none>
I20260619 09:05:17.432583 23201 fs_manager.cc:359] Metadata directory not provided
I20260619 09:05:17.432638 23201 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 09:05:17.432756 23201 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 09:05:17.433736 23201 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781859658134510-392-0/minicluster-data/master-2/data/instance:
uuid: "9db548801626401182f4232836846960"
format_stamp: "Formatted at 2026-06-19 09:05:17 on dist-test-slave-mtdr"
server_key: "0e97aa677e61f90126fc1dadd9f2cc96"
server_key_iv: "3b756ca1c0922eed1fbecbf9a6de9243"
server_key_version: "encryptionkey@0"
I20260619 09:05:17.434039 23201 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781859658134510-392-0/minicluster-data/master-2/wal/instance:
uuid: "9db548801626401182f4232836846960"
format_stamp: "Formatted at 2026-06-19 09:05:17 on dist-test-slave-mtdr"
server_key: "0e97aa677e61f90126fc1dadd9f2cc96"
server_key_iv: "3b756ca1c0922eed1fbecbf9a6de9243"
server_key_version: "encryptionkey@0"
I20260619 09:05:17.435281 23201 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260619 09:05:17.435997 23216 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 09:05:17.436197 23201 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260619 09:05:17.436283 23201 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781859658134510-392-0/minicluster-data/master-2/data,/tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781859658134510-392-0/minicluster-data/master-2/wal
uuid: "9db548801626401182f4232836846960"
format_stamp: "Formatted at 2026-06-19 09:05:17 on dist-test-slave-mtdr"
server_key: "0e97aa677e61f90126fc1dadd9f2cc96"
server_key_iv: "3b756ca1c0922eed1fbecbf9a6de9243"
server_key_version: "encryptionkey@0"
I20260619 09:05:17.436342 23201 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781859658134510-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781859658134510-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781859658134510-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 09:05:17.440255 23119 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40739 } attempt: 1
I20260619 09:05:17.440663 23121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:05:17.440541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40739 (local address 127.0.0.1:58876)
0619 09:05:17.440603 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0619 09:05:17.440635 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260619 09:05:17.440972 23121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:05:17.440887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40739 (local address 127.0.0.1:58878)
0619 09:05:17.440921 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0619 09:05:17.440946 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
W20260619 09:05:17.441048 23119 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:40739: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40739: connect: Connection refused (error 111)
I20260619 09:05:17.450977 23194 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40739 } attempt: 2
I20260619 09:05:17.451319 23196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:05:17.451219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40739 (local address 127.0.0.1:58884)
0619 09:05:17.451266 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0619 09:05:17.451292 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260619 09:05:17.451680 23196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:05:17.451560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40739 (local address 127.0.0.1:58898)
0619 09:05:17.451627 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0619 09:05:17.451658 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260619 09:05:17.451828 23194 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:40739: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40739: connect: Connection refused (error 111)
I20260619 09:05:17.466282 23201 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 09:05:17.468303 23201 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 09:05:17.468426 23201 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 09:05:17.472043 23201 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:40739
I20260619 09:05:17.472083 23268 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:40739 every 8 connection(s)
I20260619 09:05:17.472381 23201 server_base.cc:1166] Dumped server information to /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781859658134510-392-0/minicluster-data/master-2/data/info.pb
I20260619 09:05:17.473277 23269 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 09:05:17.474720 23269 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36297 } has no permanent_uuid. Determining permanent_uuid...
I20260619 09:05:17.475678 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task4ez5NB/build/release/bin/kudu as pid 23201
I20260619 09:05:17.475776 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task4ez5NB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781859658134510-392-0/minicluster-data/master-2/wal/instance
I20260619 09:05:17.475951 392 external_mini_cluster.cc:1428] Setting key 24bd804d544bd32b0cd63787f3d8e6bc
Jun 19 09:05:17 dist-test-slave-mtdr krb5kdc[23038](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859917, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Jun 19 09:05:17 dist-test-slave-mtdr krb5kdc[23038](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859917, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260619 09:05:17.483743 23122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:05:17.475363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48182 (local address 127.0.98.62:36297)
0619 09:05:17.475413 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0619 09:05:17.475415 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:05:17.475635 (+ 220us) server_negotiation.cc:407] Connection header received
0619 09:05:17.476152 (+ 517us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:05:17.476153 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:05:17.476163 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:05:17.476220 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:05:17.476909 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:05:17.477524 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:05:17.478751 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:05:17.479075 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:05:17.479151 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:05:17.481207 (+ 2056us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:05:17.481209 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:05:17.481210 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:05:17.481217 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:05:17.482509 (+ 1292us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:05:17.482905 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:05:17.482906 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:05:17.482907 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:05:17.482960 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:05:17.483166 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:05:17.483167 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:05:17.483167 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:05:17.483273 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:05:17.483326 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0619 09:05:17.483571 (+ 245us) server_negotiation.cc:299] Negotiation successful
0619 09:05:17.483643 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260619 09:05:17.483884 23271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:05:17.475359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36297 (local address 127.0.0.1:48182)
0619 09:05:17.475521 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0619 09:05:17.475556 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0619 09:05:17.476060 (+ 504us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:05:17.476440 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:05:17.476444 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:05:17.476655 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0619 09:05:17.476821 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:05:17.476825 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:05:17.477594 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:05:17.477595 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:05:17.478655 (+ 1060us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:05:17.478657 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:05:17.479185 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:05:17.479186 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:05:17.479310 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:05:17.479748 (+ 438us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 09:05:17.479755 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0619 09:05:17.481127 (+ 1372us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 09:05:17.482582 (+ 1455us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:05:17.482589 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:05:17.482591 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:05:17.482841 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:05:17.483010 (+ 169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:05:17.483010 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:05:17.483011 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:05:17.483113 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:05:17.483326 (+ 213us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 09:05:17.483332 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 09:05:17.483460 (+ 128us) client_negotiation.cc:769] Sending connection context
0619 09:05:17.483544 (+ 84us) client_negotiation.cc:240] Negotiation successful
0619 09:05:17.483678 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"mutex_wait_us":209,"thread_start_us":72,"threads_started":1}
I20260619 09:05:17.484398 23269 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41743 } has no permanent_uuid. Determining permanent_uuid...
I20260619 09:05:17.484516 23273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:05:17.476265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58914 (local address 127.0.98.60:40739)
0619 09:05:17.476410 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0619 09:05:17.476415 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 09:05:17.476465 (+ 50us) server_negotiation.cc:407] Connection header received
0619 09:05:17.476522 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:05:17.476525 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:05:17.476548 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:05:17.476629 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:05:17.477274 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:05:17.478269 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:05:17.479149 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:05:17.479550 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:05:17.479603 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:05:17.481920 (+ 2317us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:05:17.481928 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:05:17.481928 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:05:17.481934 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:05:17.483293 (+ 1359us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:05:17.483593 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:05:17.483596 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:05:17.483596 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:05:17.483645 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:05:17.483850 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:05:17.483851 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:05:17.483851 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:05:17.484006 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:05:17.484149 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0619 09:05:17.484314 (+ 165us) server_negotiation.cc:299] Negotiation successful
0619 09:05:17.484390 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":82,"threads_started":1}
Jun 19 09:05:17 dist-test-slave-mtdr krb5kdc[23038](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859917, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Jun 19 09:05:17 dist-test-slave-mtdr krb5kdc[23038](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859917, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 09:05:17.490483 23271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:05:17.484621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41743 (local address 127.0.0.1:51458)
0619 09:05:17.484661 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0619 09:05:17.484669 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0619 09:05:17.484697 (+ 28us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:05:17.485014 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:05:17.485015 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:05:17.485140 (+ 125us) client_negotiation.cc:189] Negotiated authn=SASL
0619 09:05:17.485174 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:05:17.485175 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:05:17.485744 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:05:17.485745 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:05:17.486271 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:05:17.486273 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:05:17.486596 (+ 323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:05:17.486597 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:05:17.486651 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:05:17.487014 (+ 363us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 09:05:17.487018 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 09:05:17.488089 (+ 1071us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 09:05:17.489593 (+ 1504us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:05:17.489594 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:05:17.489594 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:05:17.489743 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:05:17.489916 (+ 173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:05:17.489917 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:05:17.489917 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:05:17.489971 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:05:17.490225 (+ 254us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 09:05:17.490226 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 09:05:17.490269 (+ 43us) client_negotiation.cc:769] Sending connection context
0619 09:05:17.490337 (+ 68us) client_negotiation.cc:240] Negotiation successful
0619 09:05:17.490410 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260619 09:05:17.490499 23198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:05:17.484856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51458 (local address 127.0.98.61:41743)
0619 09:05:17.484916 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0619 09:05:17.484918 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:05:17.484926 (+ 8us) server_negotiation.cc:407] Connection header received
0619 09:05:17.484948 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:05:17.484949 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:05:17.484958 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:05:17.485007 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:05:17.485244 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:05:17.485680 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:05:17.486352 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:05:17.486541 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:05:17.486587 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:05:17.488164 (+ 1577us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:05:17.488165 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:05:17.488165 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:05:17.488172 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:05:17.489524 (+ 1352us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:05:17.489813 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:05:17.489814 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:05:17.489815 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:05:17.489866 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:05:17.490037 (+ 171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:05:17.490038 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:05:17.490039 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:05:17.490167 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:05:17.490228 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0619 09:05:17.490357 (+ 129us) server_negotiation.cc:299] Negotiation successful
0619 09:05:17.490413 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260619 09:05:17.490844 23269 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40739 } has no permanent_uuid. Determining permanent_uuid...
I20260619 09:05:17.492431 23122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:05:17.485198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48196 (local address 127.0.98.62:36297)
0619 09:05:17.485773 (+ 575us) server_negotiation.cc:206] Beginning negotiation
0619 09:05:17.485775 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:05:17.485782 (+ 7us) server_negotiation.cc:407] Connection header received
0619 09:05:17.485798 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:05:17.485799 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:05:17.485808 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:05:17.485859 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:05:17.486239 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:05:17.486775 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:05:17.487437 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:05:17.487607 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:05:17.487652 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:05:17.489790 (+ 2138us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:05:17.489791 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:05:17.489791 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:05:17.489797 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:05:17.491487 (+ 1690us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:05:17.491813 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:05:17.491814 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:05:17.491814 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:05:17.491857 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:05:17.492009 (+ 152us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:05:17.492010 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:05:17.492010 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:05:17.492089 (+ 79us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:05:17.492216 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0619 09:05:17.492277 (+ 61us) server_negotiation.cc:299] Negotiation successful
0619 09:05:17.492344 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":556}
Jun 19 09:05:17 dist-test-slave-mtdr krb5kdc[23038](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859917, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Jun 19 09:05:17 dist-test-slave-mtdr krb5kdc[23038](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859917, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260619 09:05:17.496994 23271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:05:17.491052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40739 (local address 127.0.0.1:58928)
0619 09:05:17.491133 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0619 09:05:17.491142 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 09:05:17.491187 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:05:17.491288 (+ 101us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:05:17.491288 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:05:17.491410 (+ 122us) client_negotiation.cc:189] Negotiated authn=SASL
0619 09:05:17.491441 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:05:17.491442 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:05:17.492201 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:05:17.492202 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:05:17.492793 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:05:17.492796 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:05:17.493189 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:05:17.493190 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:05:17.493255 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:05:17.493531 (+ 276us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 09:05:17.493535 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 09:05:17.494496 (+ 961us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 09:05:17.495731 (+ 1235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:05:17.495732 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:05:17.495733 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:05:17.495873 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:05:17.496016 (+ 143us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:05:17.496016 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:05:17.496017 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:05:17.496065 (+ 48us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:05:17.496708 (+ 643us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 09:05:17.496708 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 09:05:17.496753 (+ 45us) client_negotiation.cc:769] Sending connection context
0619 09:05:17.496810 (+ 57us) client_negotiation.cc:240] Negotiation successful
0619 09:05:17.496896 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":69}
I20260619 09:05:17.497058 23273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:05:17.491098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58928 (local address 127.0.98.60:40739)
0619 09:05:17.491131 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0619 09:05:17.491133 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:05:17.491174 (+ 41us) server_negotiation.cc:407] Connection header received
0619 09:05:17.491237 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:05:17.491238 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:05:17.491245 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:05:17.491281 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:05:17.491495 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:05:17.492132 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:05:17.492868 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:05:17.493125 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:05:17.493191 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:05:17.494563 (+ 1372us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:05:17.494564 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:05:17.494564 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:05:17.494570 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:05:17.495660 (+ 1090us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:05:17.495925 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:05:17.495926 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:05:17.495927 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:05:17.495970 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:05:17.496547 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:05:17.496549 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:05:17.496549 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:05:17.496650 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:05:17.496706 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0619 09:05:17.496835 (+ 129us) server_negotiation.cc:299] Negotiation successful
0619 09:05:17.496940 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":48,"server-negotiator.queue_time_us":16}
I20260619 09:05:17.498378 23269 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9db548801626401182f4232836846960: Bootstrap starting.
I20260619 09:05:17.499220 23269 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9db548801626401182f4232836846960: Neither blocks nor log segments found. Creating new log.
I20260619 09:05:17.499318 23198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:05:17.493019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51460 (local address 127.0.98.61:41743)
0619 09:05:17.493054 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0619 09:05:17.493055 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 09:05:17.493062 (+ 7us) server_negotiation.cc:407] Connection header received
0619 09:05:17.493079 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:05:17.493080 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:05:17.493088 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:05:17.493132 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:05:17.493625 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:05:17.494033 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:05:17.494661 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:05:17.494951 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:05:17.495004 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:05:17.496817 (+ 1813us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:05:17.496818 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:05:17.496818 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:05:17.496825 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:05:17.498290 (+ 1465us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:05:17.498603 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:05:17.498604 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:05:17.498604 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:05:17.498658 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:05:17.498810 (+ 152us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:05:17.498811 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:05:17.498828 (+ 17us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:05:17.498934 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:05:17.499068 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0619 09:05:17.499135 (+ 67us) server_negotiation.cc:299] Negotiation successful
0619 09:05:17.499196 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260619 09:05:17.499541 23269 log.cc:824] T 00000000000000000000000000000000 P 9db548801626401182f4232836846960: Log is configured to *not* fsync() on all Append() calls
I20260619 09:05:17.500321 23269 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9db548801626401182f4232836846960: No bootstrap required, opened a new log
I20260619 09:05:17.500607 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 09:05:17.502064 23269 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9db548801626401182f4232836846960 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b38d0e4b903340a595f25851b5eefdf5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36297 } } peers { permanent_uuid: "9ef5d197ff2c472690a74e8b1809b29a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41743 } } peers { permanent_uuid: "9db548801626401182f4232836846960" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40739 } }
I20260619 09:05:17.502223 23269 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9db548801626401182f4232836846960 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9db548801626401182f4232836846960, State: Initialized, Role: FOLLOWER
I20260619 09:05:17.502308 23269 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9db548801626401182f4232836846960 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b38d0e4b903340a595f25851b5eefdf5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36297 } } peers { permanent_uuid: "9ef5d197ff2c472690a74e8b1809b29a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41743 } } peers { permanent_uuid: "9db548801626401182f4232836846960" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40739 } }
I20260619 09:05:17.501634 23093 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "b38d0e4b903340a595f25851b5eefdf5"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48196
I20260619 09:05:17.502590 23275 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9db548801626401182f4232836846960 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b38d0e4b903340a595f25851b5eefdf5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36297 } } peers { permanent_uuid: "9ef5d197ff2c472690a74e8b1809b29a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41743 } } peers { permanent_uuid: "9db548801626401182f4232836846960" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40739 } } }
I20260619 09:05:17.502645 23269 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9db548801626401182f4232836846960 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 09:05:17.502703 23275 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9db548801626401182f4232836846960 [sys.catalog]: This master's current role is: FOLLOWER
W20260619 09:05:17.504269 23286 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9db548801626401182f4232836846960: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 09:05:17.504350 23286 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 09:05:17.504522 23201 master_runner.cc:186] Error getting master registration for 127.0.98.62:36297: OK, instance_id { permanent_uuid: "b38d0e4b903340a595f25851b5eefdf5" instance_seqno: 1781859917175961 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 09:05:17.507150 23201 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 09:05:17.516140 23119 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40739 } attempt: 2
Jun 19 09:05:17 dist-test-slave-mtdr krb5kdc[23038](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859917, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260619 09:05:17.522998 23273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:05:17.516494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58944 (local address 127.0.98.60:40739)
0619 09:05:17.516549 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0619 09:05:17.516551 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:05:17.516561 (+ 10us) server_negotiation.cc:407] Connection header received
0619 09:05:17.516607 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:05:17.516607 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:05:17.516616 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:05:17.516666 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:05:17.516966 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:05:17.517695 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:05:17.518641 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:05:17.518975 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:05:17.519035 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:05:17.520763 (+ 1728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:05:17.520764 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:05:17.520765 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:05:17.520770 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:05:17.521928 (+ 1158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:05:17.522268 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:05:17.522269 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:05:17.522269 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:05:17.522309 (+ 40us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:05:17.522488 (+ 179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:05:17.522489 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:05:17.522489 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:05:17.522606 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:05:17.522665 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0619 09:05:17.522841 (+ 176us) server_negotiation.cc:299] Negotiation successful
0619 09:05:17.522911 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260619 09:05:17.523079 23121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:05:17.516444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40739 (local address 127.0.0.1:58944)
0619 09:05:17.516501 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0619 09:05:17.516513 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 09:05:17.516560 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:05:17.516685 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:05:17.516686 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:05:17.516856 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0619 09:05:17.516903 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:05:17.516905 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:05:17.517773 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:05:17.517774 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:05:17.518572 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:05:17.518574 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:05:17.519043 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:05:17.519044 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:05:17.519097 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:05:17.519501 (+ 404us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 09:05:17.519506 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 09:05:17.520670 (+ 1164us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 09:05:17.521999 (+ 1329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:05:17.522000 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:05:17.522001 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:05:17.522210 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:05:17.522370 (+ 160us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:05:17.522370 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:05:17.522371 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:05:17.522432 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:05:17.522677 (+ 245us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 09:05:17.522678 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 09:05:17.522734 (+ 56us) client_negotiation.cc:769] Sending connection context
0619 09:05:17.522791 (+ 57us) client_negotiation.cc:240] Negotiation successful
0619 09:05:17.522995 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21}
I20260619 09:05:17.524405 23119 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b38d0e4b903340a595f25851b5eefdf5: Bootstrap starting.
I20260619 09:05:17.524992 23119 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b38d0e4b903340a595f25851b5eefdf5: Neither blocks nor log segments found. Creating new log.
I20260619 09:05:17.525246 23119 log.cc:824] T 00000000000000000000000000000000 P b38d0e4b903340a595f25851b5eefdf5: Log is configured to *not* fsync() on all Append() calls
I20260619 09:05:17.525918 23119 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b38d0e4b903340a595f25851b5eefdf5: No bootstrap required, opened a new log
I20260619 09:05:17.527241 23119 raft_consensus.cc:348] T 00000000000000000000000000000000 P b38d0e4b903340a595f25851b5eefdf5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b38d0e4b903340a595f25851b5eefdf5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36297 } } peers { permanent_uuid: "9ef5d197ff2c472690a74e8b1809b29a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41743 } } peers { permanent_uuid: "9db548801626401182f4232836846960" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40739 } }
I20260619 09:05:17.527376 23119 raft_consensus.cc:729] T 00000000000000000000000000000000 P b38d0e4b903340a595f25851b5eefdf5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b38d0e4b903340a595f25851b5eefdf5, State: Initialized, Role: FOLLOWER
I20260619 09:05:17.527462 23119 consensus_queue.cc:260] T 00000000000000000000000000000000 P b38d0e4b903340a595f25851b5eefdf5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b38d0e4b903340a595f25851b5eefdf5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36297 } } peers { permanent_uuid: "9ef5d197ff2c472690a74e8b1809b29a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41743 } } peers { permanent_uuid: "9db548801626401182f4232836846960" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40739 } }
I20260619 09:05:17.527786 23119 sys_catalog.cc:564] T 00000000000000000000000000000000 P b38d0e4b903340a595f25851b5eefdf5 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 09:05:17.527761 23289 sys_catalog.cc:455] T 00000000000000000000000000000000 P b38d0e4b903340a595f25851b5eefdf5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b38d0e4b903340a595f25851b5eefdf5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36297 } } peers { permanent_uuid: "9ef5d197ff2c472690a74e8b1809b29a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41743 } } peers { permanent_uuid: "9db548801626401182f4232836846960" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40739 } } }
I20260619 09:05:17.527860 23289 sys_catalog.cc:458] T 00000000000000000000000000000000 P b38d0e4b903340a595f25851b5eefdf5 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 09:05:17.527948 23093 raft_consensus.cc:482] T 00000000000000000000000000000000 P b38d0e4b903340a595f25851b5eefdf5 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260619 09:05:17.528033 23093 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b38d0e4b903340a595f25851b5eefdf5 [term 0 FOLLOWER]: Advancing to term 1
I20260619 09:05:17.528870 23093 raft_consensus.cc:504] T 00000000000000000000000000000000 P b38d0e4b903340a595f25851b5eefdf5 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b38d0e4b903340a595f25851b5eefdf5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36297 } } peers { permanent_uuid: "9ef5d197ff2c472690a74e8b1809b29a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41743 } } peers { permanent_uuid: "9db548801626401182f4232836846960" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40739 } }
W20260619 09:05:17.529659 23300 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b38d0e4b903340a595f25851b5eefdf5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 09:05:17.529709 23300 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 09:05:17.529783 23243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b38d0e4b903340a595f25851b5eefdf5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "9db548801626401182f4232836846960"
I20260619 09:05:17.529913 23243 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9db548801626401182f4232836846960 [term 0 FOLLOWER]: Advancing to term 1
I20260619 09:05:17.529315 23093 leader_election.cc:288] T 00000000000000000000000000000000 P b38d0e4b903340a595f25851b5eefdf5 [CANDIDATE]: Term 1 election: Requested vote from peers 9ef5d197ff2c472690a74e8b1809b29a (127.0.98.61:41743), 9db548801626401182f4232836846960 (127.0.98.60:40739)
I20260619 09:05:17.531194 23243 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 9db548801626401182f4232836846960 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b38d0e4b903340a595f25851b5eefdf5 in term 1.
I20260619 09:05:17.530659 23051 master_runner.cc:186] Error getting master registration for 127.0.98.61:41743: OK, instance_id { permanent_uuid: "9ef5d197ff2c472690a74e8b1809b29a" instance_seqno: 1781859917310588 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 09:05:17.532841 23051 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 09:05:17.532976 23067 leader_election.cc:302] T 00000000000000000000000000000000 P b38d0e4b903340a595f25851b5eefdf5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9db548801626401182f4232836846960, b38d0e4b903340a595f25851b5eefdf5; no voters:
I20260619 09:05:17.533079 23289 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b38d0e4b903340a595f25851b5eefdf5 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 09:05:17.533234 23289 raft_consensus.cc:686] T 00000000000000000000000000000000 P b38d0e4b903340a595f25851b5eefdf5 [term 1 LEADER]: Becoming Leader. State: Replica: b38d0e4b903340a595f25851b5eefdf5, State: Running, Role: LEADER
I20260619 09:05:17.533310 23289 consensus_queue.cc:237] T 00000000000000000000000000000000 P b38d0e4b903340a595f25851b5eefdf5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b38d0e4b903340a595f25851b5eefdf5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36297 } } peers { permanent_uuid: "9ef5d197ff2c472690a74e8b1809b29a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41743 } } peers { permanent_uuid: "9db548801626401182f4232836846960" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40739 } }
I20260619 09:05:17.533699 23289 sys_catalog.cc:455] T 00000000000000000000000000000000 P b38d0e4b903340a595f25851b5eefdf5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b38d0e4b903340a595f25851b5eefdf5. Latest consensus state: current_term: 1 leader_uuid: "b38d0e4b903340a595f25851b5eefdf5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b38d0e4b903340a595f25851b5eefdf5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36297 } } peers { permanent_uuid: "9ef5d197ff2c472690a74e8b1809b29a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41743 } } peers { permanent_uuid: "9db548801626401182f4232836846960" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40739 } } }
I20260619 09:05:17.533778 23289 sys_catalog.cc:458] T 00000000000000000000000000000000 P b38d0e4b903340a595f25851b5eefdf5 [sys.catalog]: This master's current role is: LEADER
I20260619 09:05:17.534991 23308 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 09:05:17.535172 23308 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 09:05:17.542958 23194 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40739 } attempt: 3
I20260619 09:05:17.544377 23122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:05:17.534661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48208 (local address 127.0.98.62:36297)
0619 09:05:17.534717 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0619 09:05:17.534718 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 09:05:17.536254 (+ 1536us) server_negotiation.cc:407] Connection header received
0619 09:05:17.536336 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:05:17.536337 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:05:17.536347 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:05:17.536373 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:05:17.537006 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:05:17.537679 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:05:17.538614 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:05:17.538930 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:05:17.538969 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:05:17.540659 (+ 1690us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:05:17.540661 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:05:17.540661 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:05:17.540668 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:05:17.542651 (+ 1983us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:05:17.543314 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:05:17.543315 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:05:17.543316 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:05:17.543388 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:05:17.543675 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:05:17.543676 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:05:17.543676 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:05:17.543812 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:05:17.543896 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0619 09:05:17.544217 (+ 321us) server_negotiation.cc:299] Negotiation successful
0619 09:05:17.544287 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260619 09:05:17.544902 23243 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9db548801626401182f4232836846960 [term 1 FOLLOWER]: Refusing update from remote peer b38d0e4b903340a595f25851b5eefdf5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 09:05:17.545545 23302 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b38d0e4b903340a595f25851b5eefdf5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9db548801626401182f4232836846960" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40739 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 09:05:17.546451 23198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:05:17.533912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51474 (local address 127.0.98.61:41743)
0619 09:05:17.533956 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0619 09:05:17.533958 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:05:17.534628 (+ 670us) server_negotiation.cc:407] Connection header received
0619 09:05:17.534762 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:05:17.534763 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:05:17.534785 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:05:17.534835 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:05:17.536124 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:05:17.536855 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:05:17.538233 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:05:17.538976 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:05:17.539010 (+ 34us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:05:17.541076 (+ 2066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:05:17.541078 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:05:17.541078 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:05:17.541096 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:05:17.542651 (+ 1555us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:05:17.545314 (+ 2663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:05:17.545315 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:05:17.545315 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:05:17.545385 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:05:17.545782 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:05:17.545783 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:05:17.545784 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:05:17.545927 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:05:17.545991 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0619 09:05:17.546289 (+ 298us) server_negotiation.cc:299] Negotiation successful
0619 09:05:17.546368 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260619 09:05:17.546917 23275 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9db548801626401182f4232836846960 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b38d0e4b903340a595f25851b5eefdf5. Latest consensus state: current_term: 1 leader_uuid: "b38d0e4b903340a595f25851b5eefdf5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b38d0e4b903340a595f25851b5eefdf5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36297 } } peers { permanent_uuid: "9ef5d197ff2c472690a74e8b1809b29a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41743 } } peers { permanent_uuid: "9db548801626401182f4232836846960" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40739 } } }
I20260619 09:05:17.547004 23275 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9db548801626401182f4232836846960 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 09:05:17.548827 23302 sys_catalog.cc:455] T 00000000000000000000000000000000 P b38d0e4b903340a595f25851b5eefdf5 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b38d0e4b903340a595f25851b5eefdf5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b38d0e4b903340a595f25851b5eefdf5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36297 } } peers { permanent_uuid: "9ef5d197ff2c472690a74e8b1809b29a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41743 } } peers { permanent_uuid: "9db548801626401182f4232836846960" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40739 } } }
I20260619 09:05:17.548936 23302 sys_catalog.cc:458] T 00000000000000000000000000000000 P b38d0e4b903340a595f25851b5eefdf5 [sys.catalog]: This master's current role is: LEADER
I20260619 09:05:17.549000 23275 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9db548801626401182f4232836846960 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b38d0e4b903340a595f25851b5eefdf5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b38d0e4b903340a595f25851b5eefdf5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36297 } } peers { permanent_uuid: "9ef5d197ff2c472690a74e8b1809b29a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41743 } } peers { permanent_uuid: "9db548801626401182f4232836846960" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40739 } } }
I20260619 09:05:17.549068 23275 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9db548801626401182f4232836846960 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 09:05:17.549222 23311 mvcc.cc:205] Tried to move back new op lower bound from 7298498222258532352 to 7298498222216953856. Current Snapshot: MvccSnapshot[applied={T|T < 7298498222258532352}]
I20260619 09:05:17.550200 23314 mvcc.cc:205] Tried to move back new op lower bound from 7298498222258532352 to 7298498222216953856. Current Snapshot: MvccSnapshot[applied={T|T < 7298498222258532352}]
I20260619 09:05:17.555230 23308 catalog_manager.cc:1347] Generated new cluster ID: 5c678e0058ac4545bca2ceabd191c17b
I20260619 09:05:17.555277 23308 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 09:05:17.555979 23273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:05:17.536014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58950 (local address 127.0.98.60:40739)
0619 09:05:17.536048 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0619 09:05:17.536049 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 09:05:17.543988 (+ 7939us) server_negotiation.cc:407] Connection header received
0619 09:05:17.544076 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:05:17.544077 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:05:17.544087 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:05:17.544115 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:05:17.547774 (+ 3659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:05:17.548431 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:05:17.550100 (+ 1669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:05:17.550366 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:05:17.550421 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:05:17.552599 (+ 2178us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:05:17.552600 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:05:17.552600 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:05:17.552607 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:05:17.554070 (+ 1463us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:05:17.554613 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:05:17.554614 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:05:17.554614 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:05:17.554683 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:05:17.555264 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:05:17.555265 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:05:17.555265 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:05:17.555380 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:05:17.555452 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0619 09:05:17.555828 (+ 376us) server_negotiation.cc:299] Negotiation successful
0619 09:05:17.555895 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
Jun 19 09:05:17 dist-test-slave-mtdr krb5kdc[23038](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781859917, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260619 09:05:17.560573 23196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:05:17.543165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40739 (local address 127.0.0.1:58962)
0619 09:05:17.543242 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0619 09:05:17.543252 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 09:05:17.543284 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 09:05:17.550419 (+ 7135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 09:05:17.550420 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 09:05:17.550558 (+ 138us) client_negotiation.cc:189] Negotiated authn=SASL
0619 09:05:17.550596 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:05:17.550598 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:05:17.551734 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:05:17.551735 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:05:17.552498 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 09:05:17.552500 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 09:05:17.554642 (+ 2142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 09:05:17.554643 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 09:05:17.554710 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:05:17.555048 (+ 338us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 09:05:17.555052 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 09:05:17.556401 (+ 1349us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 09:05:17.558984 (+ 2583us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:05:17.558984 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:05:17.558985 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:05:17.559162 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:05:17.559908 (+ 746us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 09:05:17.559909 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 09:05:17.559909 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0619 09:05:17.559967 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 09:05:17.560291 (+ 324us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 09:05:17.560292 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 09:05:17.560334 (+ 42us) client_negotiation.cc:769] Sending connection context
0619 09:05:17.560393 (+ 59us) client_negotiation.cc:240] Negotiation successful
0619 09:05:17.560482 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62}
I20260619 09:05:17.560803 23313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:05:17.543373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58962 (local address 127.0.98.60:40739)
0619 09:05:17.550292 (+ 6919us) server_negotiation.cc:206] Beginning negotiation
0619 09:05:17.550294 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:05:17.550301 (+ 7us) server_negotiation.cc:407] Connection header received
0619 09:05:17.550316 (+ 15us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:05:17.550317 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:05:17.550326 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:05:17.550919 (+ 593us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:05:17.550950 (+ 31us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:05:17.551661 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:05:17.554143 (+ 2482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:05:17.554453 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:05:17.554848 (+ 395us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:05:17.556582 (+ 1734us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 09:05:17.556583 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 09:05:17.556584 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 09:05:17.556590 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0619 09:05:17.558921 (+ 2331us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:05:17.559749 (+ 828us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:05:17.559750 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:05:17.559750 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:05:17.559810 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 09:05:17.560035 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 09:05:17.560036 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 09:05:17.560036 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 09:05:17.560233 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 09:05:17.560300 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0619 09:05:17.560439 (+ 139us) server_negotiation.cc:299] Negotiation successful
0619 09:05:17.560528 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6901,"thread_start_us":89,"threads_started":1}
I20260619 09:05:17.562202 23194 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ef5d197ff2c472690a74e8b1809b29a: Bootstrap starting.
I20260619 09:05:17.562996 23194 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9ef5d197ff2c472690a74e8b1809b29a: Neither blocks nor log segments found. Creating new log.
I20260619 09:05:17.563287 23194 log.cc:824] T 00000000000000000000000000000000 P 9ef5d197ff2c472690a74e8b1809b29a: Log is configured to *not* fsync() on all Append() calls
I20260619 09:05:17.571516 23194 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ef5d197ff2c472690a74e8b1809b29a: No bootstrap required, opened a new log
I20260619 09:05:17.573437 23194 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9ef5d197ff2c472690a74e8b1809b29a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b38d0e4b903340a595f25851b5eefdf5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36297 } } peers { permanent_uuid: "9ef5d197ff2c472690a74e8b1809b29a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41743 } } peers { permanent_uuid: "9db548801626401182f4232836846960" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40739 } }
I20260619 09:05:17.573565 23194 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9ef5d197ff2c472690a74e8b1809b29a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ef5d197ff2c472690a74e8b1809b29a, State: Initialized, Role: FOLLOWER
I20260619 09:05:17.573642 23194 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9ef5d197ff2c472690a74e8b1809b29a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b38d0e4b903340a595f25851b5eefdf5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36297 } } peers { permanent_uuid: "9ef5d197ff2c472690a74e8b1809b29a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41743 } } peers { permanent_uuid: "9db548801626401182f4232836846960" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40739 } }
I20260619 09:05:17.573969 23194 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9ef5d197ff2c472690a74e8b1809b29a [sys.catalog]: configured and running, proceeding with master startup.
I20260619 09:05:17.573998 23168 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b38d0e4b903340a595f25851b5eefdf5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "9ef5d197ff2c472690a74e8b1809b29a"
I20260619 09:05:17.574092 23168 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9ef5d197ff2c472690a74e8b1809b29a [term 0 FOLLOWER]: Advancing to term 1
I20260619 09:05:17.574935 23168 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 9ef5d197ff2c472690a74e8b1809b29a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b38d0e4b903340a595f25851b5eefdf5 in term 1.
I20260619 09:05:17.575133 23167 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9ef5d197ff2c472690a74e8b1809b29a [term 1 FOLLOWER]: Refusing update from remote peer b38d0e4b903340a595f25851b5eefdf5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 09:05:17.575541 23289 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b38d0e4b903340a595f25851b5eefdf5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ef5d197ff2c472690a74e8b1809b29a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41743 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 09:05:17.573935 23319 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ef5d197ff2c472690a74e8b1809b29a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b38d0e4b903340a595f25851b5eefdf5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36297 } } peers { permanent_uuid: "9ef5d197ff2c472690a74e8b1809b29a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41743 } } peers { permanent_uuid: "9db548801626401182f4232836846960" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40739 } } }
I20260619 09:05:17.577227 23319 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ef5d197ff2c472690a74e8b1809b29a [sys.catalog]: This master's current role is: FOLLOWER
I20260619 09:05:17.577318 23319 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ef5d197ff2c472690a74e8b1809b29a [sys.catalog]: SysCatalogTable state changed. Reason: New leader b38d0e4b903340a595f25851b5eefdf5. Latest consensus state: current_term: 1 leader_uuid: "b38d0e4b903340a595f25851b5eefdf5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b38d0e4b903340a595f25851b5eefdf5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36297 } } peers { permanent_uuid: "9ef5d197ff2c472690a74e8b1809b29a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41743 } } peers { permanent_uuid: "9db548801626401182f4232836846960" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40739 } } }
I20260619 09:05:17.577497 23319 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ef5d197ff2c472690a74e8b1809b29a [sys.catalog]: This master's current role is: FOLLOWER
I20260619 09:05:17.577579 23319 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ef5d197ff2c472690a74e8b1809b29a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b38d0e4b903340a595f25851b5eefdf5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b38d0e4b903340a595f25851b5eefdf5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36297 } } peers { permanent_uuid: "9ef5d197ff2c472690a74e8b1809b29a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41743 } } peers { permanent_uuid: "9db548801626401182f4232836846960" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40739 } } }
I20260619 09:05:17.577669 23319 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ef5d197ff2c472690a74e8b1809b29a [sys.catalog]: This master's current role is: FOLLOWER
I20260619 09:05:17.578871 23332 mvcc.cc:205] Tried to move back new op lower bound from 7298498222258532352 to 7298498222216953856. Current Snapshot: MvccSnapshot[applied={T|T < 7298498222258532352}]
I20260619 09:05:17.579680 23330 catalog_manager.cc:1259] Loaded cluster ID: 5c678e0058ac4545bca2ceabd191c17b
I20260619 09:05:17.579734 23330 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 9ef5d197ff2c472690a74e8b1809b29a: loading cluster ID for follower catalog manager: success
W20260619 09:05:17.579816 23330 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 9ef5d197ff2c472690a74e8b1809b29a: acquiring CA information for follower catalog manager: Not found: root CA entry not found
W20260619 09:05:17.579849 23330 catalog_manager.cc:873] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry
I20260619 09:05:17.580960 23289 sys_catalog.cc:455] T 00000000000000000000000000000000 P b38d0e4b903340a595f25851b5eefdf5 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b38d0e4b903340a595f25851b5eefdf5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b38d0e4b903340a595f25851b5eefdf5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36297 } } peers { permanent_uuid: "9ef5d197ff2c472690a74e8b1809b29a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41743 } } peers { permanent_uuid: "9db548801626401182f4232836846960" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40739 } } }
I20260619 09:05:17.581051 23289 sys_catalog.cc:458] T 00000000000000000000000000000000 P b38d0e4b903340a595f25851b5eefdf5 [sys.catalog]: This master's current role is: LEADER
I20260619 09:05:17.582269 23308 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 09:05:17.583001 23308 catalog_manager.cc:1504] Loading token signing keys...
I20260619 09:05:17.586589 23308 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b38d0e4b903340a595f25851b5eefdf5: Generated new TSK 0
I20260619 09:05:17.586751 23308 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 09:05:17.610237 23198 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 09:05:17.608920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51488 (local address 127.0.98.61:41743)
0619 09:05:17.608967 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0619 09:05:17.608969 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:05:17.609591 (+ 622us) server_negotiation.cc:407] Connection header received
0619 09:05:17.609706 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:05:17.609708 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:05:17.609719 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:05:17.609748 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:05:17.610166 (+ 418us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51488: BlockingRecv error: recv got EOF from 127.0.0.1:51488 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
W20260619 09:05:17.610963 23122 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 09:05:17.609283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48214 (local address 127.0.98.62:36297)
0619 09:05:17.609330 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0619 09:05:17.609332 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:05:17.610461 (+ 1129us) server_negotiation.cc:407] Connection header received
0619 09:05:17.610551 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:05:17.610552 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:05:17.610561 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:05:17.610588 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:05:17.610904 (+ 316us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48214: BlockingRecv error: recv got EOF from 127.0.0.1:48214 (error 108)
Metrics: {"server-negotiator.queue_time_us":29}
W20260619 09:05:17.611950 23313 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 09:05:17.611289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58968 (local address 127.0.98.60:40739)
0619 09:05:17.611327 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0619 09:05:17.611329 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:05:17.611447 (+ 118us) server_negotiation.cc:407] Connection header received
0619 09:05:17.611530 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:05:17.611531 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:05:17.611540 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:05:17.611566 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:05:17.611895 (+ 329us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58968: BlockingRecv error: recv got EOF from 127.0.0.1:58968 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260619 09:05:17.614033 23198 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 09:05:17.613345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51492 (local address 127.0.98.61:41743)
0619 09:05:17.613382 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0619 09:05:17.613384 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:05:17.613557 (+ 173us) server_negotiation.cc:407] Connection header received
0619 09:05:17.613667 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:05:17.613668 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:05:17.613684 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:05:17.613711 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:05:17.613989 (+ 278us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51492: BlockingRecv error: recv got EOF from 127.0.0.1:51492 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260619 09:05:17.616881 23198 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 09:05:17.616178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51500 (local address 127.0.98.61:41743)
0619 09:05:17.616209 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0619 09:05:17.616211 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:05:17.616417 (+ 206us) server_negotiation.cc:407] Connection header received
0619 09:05:17.616498 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:05:17.616499 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:05:17.616508 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:05:17.616534 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:05:17.616825 (+ 291us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51500: BlockingRecv error: recv got EOF from 127.0.0.1:51500 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260619 09:05:17.620647 23198 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 09:05:17.619880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51514 (local address 127.0.98.61:41743)
0619 09:05:17.619910 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0619 09:05:17.619912 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:05:17.620185 (+ 273us) server_negotiation.cc:407] Connection header received
0619 09:05:17.620273 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:05:17.620274 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:05:17.620284 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:05:17.620310 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:05:17.620594 (+ 284us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51514: BlockingRecv error: recv got EOF from 127.0.0.1:51514 (error 108)
Metrics: {"server-negotiator.queue_time_us":15}
W20260619 09:05:17.626314 23198 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 09:05:17.625604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51524 (local address 127.0.98.61:41743)
0619 09:05:17.625647 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0619 09:05:17.625649 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:05:17.625864 (+ 215us) server_negotiation.cc:407] Connection header received
0619 09:05:17.625949 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:05:17.625950 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:05:17.625958 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:05:17.625984 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:05:17.626267 (+ 283us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51524: BlockingRecv error: recv got EOF from 127.0.0.1:51524 (error 108)
Metrics: {"server-negotiator.queue_time_us":29}
W20260619 09:05:17.636135 23198 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 09:05:17.635373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51532 (local address 127.0.98.61:41743)
0619 09:05:17.635417 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0619 09:05:17.635418 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 09:05:17.635654 (+ 236us) server_negotiation.cc:407] Connection header received
0619 09:05:17.635747 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:05:17.635748 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:05:17.635758 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:05:17.635787 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:05:17.636087 (+ 300us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51532: BlockingRecv error: recv got EOF from 127.0.0.1:51532 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260619 09:05:17.654199 23198 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 09:05:17.653281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51538 (local address 127.0.98.61:41743)
0619 09:05:17.653322 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0619 09:05:17.653324 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:05:17.653600 (+ 276us) server_negotiation.cc:407] Connection header received
0619 09:05:17.653719 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:05:17.653720 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:05:17.653732 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:05:17.653761 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:05:17.654144 (+ 383us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51538: BlockingRecv error: recv got EOF from 127.0.0.1:51538 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260619 09:05:17.688421 23198 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 09:05:17.687475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51554 (local address 127.0.98.61:41743)
0619 09:05:17.687523 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0619 09:05:17.687525 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:05:17.687829 (+ 304us) server_negotiation.cc:407] Connection header received
0619 09:05:17.687934 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:05:17.687935 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:05:17.687946 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:05:17.687975 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:05:17.688361 (+ 386us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51554: BlockingRecv error: recv got EOF from 127.0.0.1:51554 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260619 09:05:17.754351 23198 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 09:05:17.753770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51566 (local address 127.0.98.61:41743)
0619 09:05:17.753826 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0619 09:05:17.753828 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:05:17.753884 (+ 56us) server_negotiation.cc:407] Connection header received
0619 09:05:17.753945 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:05:17.753946 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:05:17.753955 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:05:17.754015 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:05:17.754303 (+ 288us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51566: BlockingRecv error: recv got EOF from 127.0.0.1:51566 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
W20260619 09:05:17.884145 23198 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 09:05:17.883481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51580 (local address 127.0.98.61:41743)
0619 09:05:17.883557 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0619 09:05:17.883560 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 09:05:17.883630 (+ 70us) server_negotiation.cc:407] Connection header received
0619 09:05:17.883702 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:05:17.883704 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:05:17.883714 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:05:17.883763 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:05:17.884079 (+ 316us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51580: BlockingRecv error: recv got EOF from 127.0.0.1:51580 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260619 09:05:18.141732 23198 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 09:05:18.141190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51594 (local address 127.0.98.61:41743)
0619 09:05:18.141250 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0619 09:05:18.141252 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:05:18.141260 (+ 8us) server_negotiation.cc:407] Connection header received
0619 09:05:18.141286 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:05:18.141287 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:05:18.141297 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:05:18.141364 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0619 09:05:18.141664 (+ 300us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51594: BlockingRecv error: recv got EOF from 127.0.0.1:51594 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
I20260619 09:05:18.504768 23286 catalog_manager.cc:1259] Loaded cluster ID: 5c678e0058ac4545bca2ceabd191c17b
I20260619 09:05:18.504839 23286 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 9db548801626401182f4232836846960: loading cluster ID for follower catalog manager: success
I20260619 09:05:18.505640 23286 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 9db548801626401182f4232836846960: acquiring CA information for follower catalog manager: success
I20260619 09:05:18.505939 23286 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 9db548801626401182f4232836846960: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 09:05:18.580947 23330 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 9ef5d197ff2c472690a74e8b1809b29a: acquiring CA information for follower catalog manager: success
I20260619 09:05:18.581312 23330 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 9ef5d197ff2c472690a74e8b1809b29a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 09:05:18.657500 23400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:05:18.654935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51608 (local address 127.0.98.61:41743)
0619 09:05:18.655052 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0619 09:05:18.655054 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:05:18.655067 (+ 13us) server_negotiation.cc:407] Connection header received
0619 09:05:18.655151 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:05:18.655152 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:05:18.655208 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:05:18.655278 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 09:05:18.655439 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:05:18.655972 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:05:18.656752 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:05:18.656951 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:05:18.657011 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:05:18.657164 (+ 153us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 09:05:18.657274 (+ 110us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 09:05:18.657347 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0619 09:05:18.657386 (+ 39us) server_negotiation.cc:299] Negotiation successful
0619 09:05:18.657408 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95,"thread_start_us":59,"threads_started":1}
I20260619 09:05:18.663261 23406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:05:18.659550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58970 (local address 127.0.98.60:40739)
0619 09:05:18.660954 (+ 1404us) server_negotiation.cc:206] Beginning negotiation
0619 09:05:18.660957 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 09:05:18.660966 (+ 9us) server_negotiation.cc:407] Connection header received
0619 09:05:18.661008 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:05:18.661009 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:05:18.661063 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:05:18.661146 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 09:05:18.661265 (+ 119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:05:18.661722 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:05:18.662466 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:05:18.662672 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:05:18.662721 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:05:18.662951 (+ 230us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 09:05:18.663041 (+ 90us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 09:05:18.663098 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0619 09:05:18.663164 (+ 66us) server_negotiation.cc:299] Negotiation successful
0619 09:05:18.663182 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1380,"thread_start_us":82,"threads_started":1}
I20260619 09:05:18.667057 23400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:05:18.664761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51618 (local address 127.0.98.61:41743)
0619 09:05:18.664895 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0619 09:05:18.664897 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 09:05:18.664998 (+ 101us) server_negotiation.cc:407] Connection header received
0619 09:05:18.665046 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:05:18.665047 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:05:18.665057 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:05:18.665103 (+ 46us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 09:05:18.665233 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:05:18.665752 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:05:18.666497 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:05:18.666669 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:05:18.666710 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:05:18.666858 (+ 148us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 09:05:18.666886 (+ 28us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 09:05:18.666921 (+ 35us) server_negotiation.cc:1035] Waiting for connection context
0619 09:05:18.666981 (+ 60us) server_negotiation.cc:299] Negotiation successful
0619 09:05:18.666996 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115}
I20260619 09:05:18.667848 23412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:05:18.664765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48224 (local address 127.0.98.62:36297)
0619 09:05:18.665034 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0619 09:05:18.665035 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 09:05:18.665116 (+ 81us) server_negotiation.cc:407] Connection header received
0619 09:05:18.665210 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:05:18.665211 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:05:18.665261 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:05:18.665325 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 09:05:18.665458 (+ 133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:05:18.666210 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:05:18.667047 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:05:18.667320 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:05:18.667384 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:05:18.667578 (+ 194us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 09:05:18.667648 (+ 70us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 09:05:18.667693 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0619 09:05:18.667748 (+ 55us) server_negotiation.cc:299] Negotiation successful
0619 09:05:18.667769 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":250,"thread_start_us":92,"threads_started":1}
I20260619 09:05:18.668419 23406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 09:05:18.664837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58986 (local address 127.0.98.60:40739)
0619 09:05:18.664874 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0619 09:05:18.664875 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0619 09:05:18.665073 (+ 198us) server_negotiation.cc:407] Connection header received
0619 09:05:18.665127 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 09:05:18.665128 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 09:05:18.665137 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 09:05:18.665182 (+ 45us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 09:05:18.665475 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:05:18.666204 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:05:18.667779 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 09:05:18.668030 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 09:05:18.668076 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 09:05:18.668215 (+ 139us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 09:05:18.668239 (+ 24us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 09:05:18.668273 (+ 34us) server_negotiation.cc:1035] Waiting for connection context
0619 09:05:18.668333 (+ 60us) server_negotiation.cc:299] Negotiation successful
0619 09:05:18.668351 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260619 09:05:18.669582 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ez5NB/build/release/bin/kudu with pid 23051
I20260619 09:05:18.674387 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ez5NB/build/release/bin/kudu with pid 23126
I20260619 09:05:18.678435 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task4ez5NB/build/release/bin/kudu with pid 23201
2026-06-19T09:05:18Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (2666 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (5394 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (260548 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST